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LETTER  OF  TRANSMITTAL 


House  of  Representatives, 
Committee  on  Science  and  Technology, 

Washington,  D.C.,  January  19,  1976. 
Hon.  Olin  E.  Teague, 

Chairman,  Committee  on  Science  and  Technology,  U.S.  House  of  Repre- 
sentatives, Washington,  D.C. 
Dear  Mr.  Chairman:  During  the  summer  of  1975  the  Subcommit- 
tee on  Science,  Research  and  Technology  held  six  days  of  oversight 
hearings  into  National  Science  Foundation  peer  review  procedures. 
These  hearings  were  part  of  the  Subcommittee's  continuing  oversight 
of  the  Foundation. 

The  following  report  of  the  Subcommittee  is  based  upon  the  analysis 

of  the  record  which  has  taken  place  since  the  hearings.  The  report 

contains  conclusions  and  recommendations  of  the  Subcommittee.  It  is 

submitted  for  the  use  of  the  Committee  on  Science  and  Technology. 

Sincerely, 

James  W.  Symington, 
Chairman,  Subcommittee  on  Science,  Research  and  Technology. 
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I.  Summary 

A.  INTRODUCTION 

The  Subcommittee  on  Science,  Research  and  Technology  of  the 
House  Committee  on  Science  and  Technology  held  six  days  of  hear- 
ings during  July  1975  as  part  of  the  Subcommittee's  continuing 
oversight  of  the  National  Science  Foundation.  This  report  stems 
from  those  hearings  and  is  based  upon  the  oral  and  written  testimony 
submitted  during  the  hearings.  The  question  on  which  the  Sub- 
committee's investigation  has  been  focused  is:  "Once  monies  have 
been  appropriated  to  the  Foundation  and  the  division  of  those  monies 
among  disciplines  and  purposes  has  been  agreed  upon,  how  does  the 
Foundation  go  about  making  individual  awards?"  This  topic  was 
appropiiate  for  several  reasons.  Foiemost  among  these  were:  Con- 
gressional and  public  concern  that  the  Foundation  might  be  supporting 
questionable  research,  spreading  belief  that  government  operations 
ought  to  be  open  to  public  scruthry,  and  concern  that  tightness  of  funds 
might  result  in  complaints  about  the  Foundation's  decision-making 
systems  from  the  scientific  community. 

The  mode  in  which  the  Foundation  provides  most  of  it*  support 
for  scientific  activity  is  to  fund  projects  diiected  by  individuals 
at  institutions.  The  Subcommittee  believes  this  is  a  sound  mode  of 
support  and  has  turned  its  attention  primarily  to  the  question  of  how 
projects  are  selected  for  funding.  The  Foundation's  selection  pro- 
cedures are  based  principally  upon  evaluations  of  project  proposals 
which  have  been  submitted  to  the  Foundation.  Evaluation  piocedures 
usually  include  an  assessment  of  the  merit  of  the  proposal  and  the 
competence  of  the  proposer  by  experts  from  outside  the  Foundation 
in  the  subject  matter  of  the  proposal.  The  experts  are  considered  peers 
of  the  proposer.  Because  of  their  review,  the  Foundation's  evaluation 
sy-tem  is  commonly  called  a  peer  review  system,  and  the  hearings 
have  been  designated  "Peer  Review  Hearings." 

In  fact,  the  Foundation  employs  a  lather  wide  vaiiety  of  evalua- 
tion procedures  tailored  to  fit  differing  types  of  programs,  so  that 
caution  is  necessary  in  deriving  01  applying  general  rules.  The  pro- 
cedures normally  include  reliance  on  staff  judgement  plus  evaluation 
by  individual  peer  reviewers,  evaluation  by  an  assembled  panel  of 
peer  reviewers,  or  both.  Other  elements  that  may  be  included  are: 
site  visits  National  Science  Board  action,  or  consultation  with  other 
funding  agencies. 

Witnesses  who  appeared  included  two  Congressmen  who  asked  to 
testify,  four  principal  Foundation  witnesses,  ten  scientific  experts  from 
outside  the  Foundation  who  were  selected  by  the  Subcommittee  for 
their  special  competence  and  viewpoints,  and  seven  other  minority 
witnesses.  The  two  Congressmen,  Rep.  John  Conlan  of  Arizona  and 
Rep.  Robert  Bauman  of  Maryland  both  had  criticized  the  Foundation 
in  matters  other  than  proposal  evaluation  before  the  hearings  and  were 
both  critical  in  their  testimony  at  the  hearings.  Foundation  witnesses 
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devoted  much  of  their  testimony  to  describing  how  the  Foundation 
makes  decisions  and  what  the  resulting  distribution  of  Foundation 
funds  and  reviewers  has  been;  they  found  fundamental  soundness  in 
existing  Foundation  procedures.  The  experts  selected  by  the  Subcom- 
mittee from  outside  the  Foundation  were  generally  supportive  of  the 
Foundation  while  making  suggestions  for  improvements.  The  seven 
other  minority  witnesses,  all  of  whom  had  been  suggested  by  Rep. 
Conlan.  were  critical  of  the  Foundation.  Philip  Handler,  President 
of  the  National  Academy  of  Sciences,  submitted  important  written 
testimony  although  he  had  been  unable  to  appear  in  person  because 
of  illness. 

B.  ISSUES 

A  wealth  of  interacting  issues  concerning  peer  review  was  considered 
by  the  Subcommittee.  The  amount  and  type  of  information  available 
for  helping  reach  judgments  varied  widely  from  issue  to  issue.  Conse- 
quently, there  is  variation  in  the  type  of  findings  and  recommendations 
the  Subcommittee  has  formulated.  The  issues  are  presented  in  chapter 
IV  and  analyzed  in  chapter  V.  Chapter  VI  contain-  the  Subcommittee's 
recommendations  organized  by  issue.  This  summary  is  directed  only 
to  selected  key  points. 

C.   PRINCIPAL  FINDINGS  AND  RECOMMENDATIONS 

In  the  Subcommittee's  judgement,  the  most  important  single  con- 
sideration of  the  Foundation's  peer  review  system  is  whether  the 
Foundation's  awards  lead  to  the  advance  of  science  in  the  most  effec- 
tive possible  way.  The  Subcommittee's  principal  findings  and  recom- 
mendations are: 

1.  Finding:  Procedures  relying  on  peer  review  for  evaluating  applications  for 
scientific  research  awards  can  lead  to  the  effective  advance  of  scientific  knowledge. 
The  National  Science  Foundation's  peer  review  evaluation  systems  appear 
basically  sound. 

Xo  method  superior  to  peer  review  has  been  found  for  judging  the 
scientific  competence  of  proposers.  Scientific  peer<  are  better  able  than 
others  to  judge  the  design  of  proposed  work,  the  importance  of  proposed 
work  to  the  scientific  field,  and  the  past  performance  of  the  proposer. 
Appropriate  peer  review  procedures  generally  lead  to  the  support  of 
proposals  in  a  high  quality  range.  Using  peer  review  procedures  the 
Foundation  has  successfully  fostered  significant  advance^  in  basic- 
science  over  the  past  25  years. 

Recommendation:  The  National  Science  Foundation  should  continue  to  use 
some  form  of  peer  review.  Congress  should  not  require  major  changes  at  this  time 
in  the  methods  used  by  the  Foundation  for  proposal  evaluation. 

2.  Finding:  Certain  problems  exist  with  the  Foundation's  peer  review  evaluation 
processes  and  related  operations.  Substantial  changes  in  the  operation  of  the 
Foundation's  peer  review  systems  have  been  proposed  which  possibly  could 
lead  to  improvements  in  the  system's  effectiveness  for  consistently  selecting  the 
best  proposals,  and  in  the  acceptability  of  the  systems  to  potential  applicants,  the 
scientific  community,  and  Congress. 

Recommendation:  Some  corrective  actions  should  be  taken  immediately  and 
certain  proposals  for  substantial  changes  should  be  thoroughly  investigated  and 
instituted  if  judged  worthy. 


D.   IMMEDIATE  ACTION 

The  recommendations  for  immediate  action  mentioned  in  the  second 
principal  recommendation,  together  with  brief  discus-ion-  of  the 
rationale  for  each,  follow. 

Recommendation:  The  Foundation  should  prepare  an  organized  compendium 
of  its  policies  in  writing  and  assure  that  it  be  widely  available. 

Examples  of  staff  action-  contrary  to  Foundation  policy  were  given 
at  the  hearings.  Although  the  Subcommittee  doe-  not  believe  such 
actions  are  widespread,  the  compendium  of  policy  and  procedure 
memoranda  submitted  by  the  Foundation,  because  of  it-  size  and  lack 
of  organization,  give-  the  Subcommittee  little  reassurance  that  the 
staff  can  stay  abreast  of  current  policies  even  with  the  best  intention-. 
The  Subcommittee  believes  that  the  scientific  community  should  be 
able  to  understand  and  stay  current  with  Foundation  policies.  At 
present  there  is  no  way  such  understanding  can  be  easily  acquired. 

Recommendation:  The  Foundation  should  devise  .systematic  methods  of  en- 
suring that  competent  reviewers  are  identified  and  used  regardless  of  institutional 
affiliation  or  geographical  location.  No  reviewer  should  be  overused  in  relation 
to  his  or  her  abilities. 

The  Subcommittee  believes  that  reviewing  should  be  widely  dis- 
tributed among  competent  potential  reviewers.  On  a  Foundation-wide 
geographic  basis,  rather  broad  distribution  appears  to  have  occurred. 
There  are,  however,  no  systematic  methods  employed  by  the  Founda- 
tion for  assuring  wide  distribution  or  determining  whether  any  given 
reviewer  has  been  too  frequently  used. 

Recommendation:  The  National  Science  Foundation  should  perform  random 
audits  of  the  decision  processes  used  for  individual  grants.  The  Foundation  should 
report  to  Congress  describing  the  system  it  uses  for  this  type  of  audit. 

Recommendation:  The  Subcommittee  on  Science,  Research  and  Technology  of 
the  House  Committee  on  Science  and  Technology  and  the  Subcommittee  on  the 
National  Science  Foundation  of  the  Senate  Committee  on  Labor  and  Public 
Welfare  should  work  with  the  Foundation  to  establish  procedures  whereby  the 
Subcommittees  can  have  effective  access  to  certain  confidential  information  of 
the  Foundation.  The  procedures  should  be  designed  to  maintain  the  highest 
degree  of  confidentiality  of  the  information  that  will  allow  effective  Congressional 
oversight. 

These  two  recommendations  address  the  widely-discussed  issue  of 
bias  in  peer  review  systems  and  incorporate  the  Subcommittee's  belief 
that  oversight  of  the  Foundation's  peer  review  system  should  be  based 
on  an  attitude  of  trust  combined  with  vigilance  over  the  system  to 
assure  that  trust  is  warranted. 

It  is  noteworthy  that  although  the  severest  critics  of  Foundation 
peer  review  have  labeled  it  "an  incestuous  buddy  system"  no  example 
of  bias  on  the  part  of  reviewers  was  mentioned  at  the  hearings. 
Conversely,  neither  was  it  demonstrated  that  the  system  is  not 
biased.  The  reasons  for  this  lack  of  surety  appear  to  be  principally  (i) 
talent  is  concentrated,  so  that  the  appearance  of  buddy  systems 
cannot  be  avoided,  but  (ii)  judgement-;  of  whether  a  reviewer  has  been 
biased  in  a  particular  instance  are  almost  wholly  subjective. 

The  recommended  audits  should  provide  a  check  of  the  relations 
among  the  Foundation  staff,  proposers,  and  reviewers  in  many  individual 
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instances,  and  the  combined  results  of  many  audits  should  help  give 
statistical  evidence  for  or  against  bias. 

Arranging  for  access  to  Foundation  information  by  the  two  Con- 
gressional committees  charged  with  authorization  of  the  Foundation's 
budget  is  necessary  to  proper  Congressional  oversight. 

Recommendation:  The  Foundation  should  include  in  each  letter  announcing  a 
funding  decision  to  an  applicant  either  a  statement  of  the  methods  and  rationale 
of  the  decision  or  a  statement  that  such  information  will  be  provided  on  request. 

Applicants  for  Foundation  awards  have  the  right  to  know  the 
methods  and  rationales  of  decisions  made  concerning  their  applica- 
tions. It  is  not  clear  that  applicants  have  had  adequate  access  to  such 
information  in  the  past. 

Recommendation:  The  Foundation  should  fill  reasonable  requests  for  data 
concerning  the  distribution  of  funds  by  the  Metallurgy  and  Materials  Section 
within  the  constraints  of  the  Foundation's  policies  concerning  the  confidentiality 
of  such  information. 

The  Subcommittee  found  that  evidence  presented  which  purported 
to  show  that  the  Metallurgy  and  Materials  Section  has  distributed 
funds  inequitably  is  highly  questionable.  The  primary  reasons  that  the 
evidence  is  questionable  are:  (1)  citation  analysis  alone  was  used  to 
produce  a  measure  of  scientific  ability,  but  citation  analysis  standing 
alone  appears  to  be  inadequate  for  that  purpose,  and  (2)  to  be  mean- 
ingful, an  analysis  of  funding  patterns  must  encompass  all  agencies 
providing  support.  Additionally,  many  issues  concerning  technical 
aspects  of  the  evidence  have  been  raised.  It  was  not  clear,  however, 
whether  the  Foundation  had  adequately  filled  requests  by  the  principal 
witness  for  information.  This  recommendation  addresses  that  point. 

E.    NATIONAL    SCIENCE    BOARD 

In  the  course  of  considering  peer  review  at  the  Foundation  it 
became  clear  that  there  are  many  important  issues  which  should  not 
be  judged  without  further  careful  study  by  persons  familiar  with  peer 
review  systems,  further  collection  of  information,  or  further  analysis 
after  the  elapse  of  a  period  of  time.  The  Subcommittee  considered 
arguments  that  Congress  ought  to  judge  these  issues  and  set  Founda- 
tion policies  but  is  convinced  it  is  preferable  to  keep  the  responsibility 
for  setting  Foundation  policies  lodged  in  the  National  Science  Board 
as  legislated  in  the  Foundation's  organic  act. 

The  Subcommittee  recommends  in  this  report  that  the  National 
Science  Board  devote  further  study  to  eight  issues.  It  is  desirable 
that  the  Board  turn  its  attention  promptly  to  these  issues.  The  Sub- 
committee will  stay  informed  of  the  Board's  progress  to  assure  that 
the  issues  are  sufficiently  addressed : 

Finding:  There  is  a  clear  need  for  firm  policy  guidance  in  the  management  of 
peer  review  at  the  National  Science  Foundation.  Experience  in  science  and  with 
the  scientific  community,  as  well  as  carefully  assembled  objective  information,  are 
essential  to  the  formation  of  sound  policies  governing  peer  review. 

Recommendation:  The  National  Science  Board  should  have  primary  respon- 
sibility for  the  establishment  of  policies  governing  peer  review  at  the  National 
Science  Foundation. 

The  National  Science  Board  should — 

1.  Study  the  support  of  innovative  research  and  report  to  Congress. 

2.  Study  the  support  of  young  scientists  and  report  to  Congress. 

3.  Study  the  funding  of  research  at  undergraduate-teaching  institutions 
without  graduate  departments  (colleges)  by  the  Foundation  and  report  to 
Congress. 


4.  Study  the  extent  to  which  the  Foundation  should  rely  on  peer  panel 
review  and  report  to  Congress. 

5.  Establish  an  interna]  Foundation  program  to  monitor  problem-  arising 
from  the  mismatch  between  the  size  of  the  scientific  community  and  the 

amount  of  Foundation  funds  available  for  support  of  that  community,  and 
should  report  periodically  to  Congr 

6.  Study  the  question  of  whether  the  National  Science  Foundation  should 
have  formal  procedures  lor  considering  appeals  of  decisions  made  on  award 
applications  and  should  report  to  Congress. 

7.  Study  the  effects  of  publication  of  the  li-t  of  reviewers  used  by  the 
Foundation  and  consider  whether  publication  of  the  list  in  a  less  aggregated 
form  might  be  desirable. 

8.  Collect  further  information  concerning  effects  on  the  peer  review  sys- 
tem of  the  level  of  confidentiality  in  which  peer  reviewers'  name-  and  ver- 
batim comments  are  held.  The  Board  should  report  the  information  and  any 
conclusions  that  may  be  drawn  from  it  to  Congress.  Further  changes  in  the 
level  of  confidentiality  of  the  Foundation--  peer  review  -y-tem  should  be 
made  slowly  if  at  all. 

F.  OPENNESS 

The  last  two  recommendations  for  Board  actions  are  part  of  an 
inclination  by  the  Subcommittee  towards  openne— 

Recommendation:  The  greatest  degree  of  openness  of  the  Foundation's  award 
decision-making  process  consonant  with  effective  proposal  evaluation  and  reason- 
able efficiency  should  be  achieved. 

The  most  thoroughly  discussed  i--ue  of  the  hearings  was  the  ques- 
tion of  the  extent  to  which  openness  of  the  decision-making  process 

can  coexist  with  the  effective  evaluation  of  award  application-  by  the 
Foundation.  Arguments  are  presented  in  some  detail  in  chapter  V  of 
this  report.  The  Subcommittee  has  concluded  that  since  (i)  the  issue 
appears  to  be  critical,  (ii)  the  Foundation  has  recently  increased  its 
degree  of  openness  and.  (iii)  in  contrast  to  the  number  of  arguments, 
hard  facts  about  the  situation  arc  few,  the  best  course  of  action  at  this 
time  is  to  evaluate  the  effect-  of  the  recent  change-  and  continue  study 
of  the  is<ue  by  the  National  Science  Board  as  indicated  above. 

Other  parts  of  the  thrust  toward-  openness  already  mentioned  are 
the  recommendation-  that  the  Foundation  should  prepare  and  dis- 
tribute an  organized  compendium  of  it-  policies  and  should  include 
infoimation  about  the  methods  and  rationale  of  it-  decisions  when 
the  decisions  are  announced  to  applicants. 

G.  CONGRESS 

From  its  study  of  peer  review  at  the  Foundation,  it  has  become  clear 
to  the  Subcommittee  that  the  role  of  Congress  also  requires  attention. 

Recommendation:  Congress  should  not  require  Congressional  review  of  indi- 
vidual research  awards. 

The  Subcommittee  believes  that  Congressional  review  of  National 
Science  Foundation  awards  would  introduce  serious  problems  into  the 
government's  support  of  science,  including:  (i)  politicization  of  the 
award  decision-making  process,  turning  it  away  from  support  of  pro- 
po-als  on  their  merit,  (ii)  less  willingness  in  the  Foundation  to 
support  innovative  ideas,  and  (iii)  lengthening  of  the  time  it  take- 
applicants  to  obtain  funding  and  proceed  with  their  research. 

Recommendation:  If  Congress  undertakes  studies  of  the  geographical  distribu- 
tion of  funds  for  scientific  research  by  any  Federal  agency  or  agencies,  such 
studies  should  explicitly  put  support  by  one  agency  in  the  context  of  total  Federal 
support. 


6 

There  is  division  of  opinion  among  members  of  the  Subcommittee 
concerning  the  desirability  of  requiring  that  the  geographical  distribu- 
tion of  National  Science  Foundation  funds  meet  some  standard  of 
evenness.  Some  argue  that  the  Foundation  should  support  the  best 
research  wherever  and  in  whatever  concentration  it  may  be  found. 
Others  believe  some  degree  of  evenness  should  be  required.  The 
Subcommittee  finds  that  rational  consideration  of  the  geographical 
distribution  of  Federal  funds  for  scientific  research  requires  that 
funding  patterns  be  considered  on  a  government-wide  basis,  including 
all  agencies  providing  significant  amounts  of  money  for  scientific 
research,  not  only  the  Foundation. 

The  Subcommittee  is,  nonetheless,  mindful  of  the  requirements  of 
Public  Law  81-507,  Section  3(e),  the  Organic  Act  of  the  Foundation, 
and  cautions  the  Foundation  that  it  has  a  special  statutory  duty  to 
avoid  undue  concentration  in  its  grant  decisions. 

Recommendation:  Congress  should  not  propose  or  acquiesce  to  reductions  in 
the  overlap  of  fields  of  basic  scientific  research  supported  by  various  Federal 
agencies  without  further  investigating  the  advantages  and  disadvantages  of  such 
overlaps. 

It  has  been  argued  that  the  existence  of  multiple  funding  sources  in 
the  Federal  government  has  been  one  of  the  great  strengths  of  Amer- 
ican science — that  the  best  way  to  prevent  over-direction  of  basic 
research  and  stifling  of  innovation  by  the  Federal  bureacracy  is  to 
ensure  that  there  is  more  than  one  source  of  support.  These  argu- 
ments deserve  careful  consideration.  This  recommendation  does  not 
mean  the  Subcommittee  favors  duplicative  research  projects. 


II.  Background 

A.  LEGISLATIVE   MANDATE 

During  the  fall  and  winter  of  1974,  the  House  of  Representatives 
Committee  on  Seience  and  Astronautics  was  reorganized.  Its  name 
became  the  Committee  on  Science  and  Technology,  its  responsibilities 
for  legislation  and  oversight  were  enlarged,  and  the  number  of  its 
subcommittees  was  increased.  The  former  Subcommittee  on  Science, 
Research  and  Development  was  renamed  to  be  the  Subcommittee 
on  Science,  Research  and  Technology.  James  Symington,  Democrat  of 
Missouri,  became  the  Subcommittee's  chairman — he  had  not  been 
chairman  earlier — and  the  size  of  the  Subcommittee  staff  doubled. 

These  changes  permitted  the  Subcommittee  to  begin  exercising  fuller 
oversight  of  the  National  Science  Foundation  than  had  been  possible 
in  former  years,  although  the  Subcommittee's  charge  regarding  the 
National  Science  Foundation  was  not  new. 

Discussions  within  the  Committee  on  Science  and  Technology  and 
between  the  Subcommittee  and  the  Foundation  during  the  spring  of 
1975  resulted  in  the  selection  of  "peer  review"  as  the  topic  for  the 
July  oversight  hearings  (authorization  hearings  on  the  Foundation's 
proposed  budget  for  Fiscal  Year  1976  and  oversight  hearings  on  the 
Foundation's  education  programs  had  been  held  by  the  Subcommittee 
in  early  1975  under  Mr.  Symington's  chairmanship). 

B.  SURROUNDING   EVENTS 

Four  factors  made  the  selection  of  peer  review  at  the  Foundation 
particularly  appropriate  as  the  topic  for  oversight  hearings:  congres- 
sional activity,  budget  concerns  of  the  public,  the  desire  for  freedom  of 
information,  and  increased  competition  for  Foundation  grants. 

1.  Congress 

Several  Congressmen  had  expressed  concern  that  the  Foundation 
seemed  to  be  supporting  useless  research.  The  most  widely  reported 
allegations  of  waste  were  voiced  by  Senator  William  Proxmire  and 
Representative  Robert  Bauman.  Representative  John  Conlan  had 
called  attention  to  what  he  said  were  abuses  of  peer  review  by  the 
Foundation.  Many  other  Congressmen,  while  not  so  outspoken,  had 
received  letters  from  their  constituents  questioning  expenditures  on 
individual  scientific  grants  These  Congressmen  wanted  to  know 
whether  the  Foundation  was  performing  competently. 

2.  Public 

The  letters  from  constituents  almost  invariably  had  resulted  from 
attention  being  drawn  in  the  press  to  titles  of  scientific  grants  that 
sounded  silly.  The  public  was  generally  concerned  with  the  size  of  the 
Federal  budget  and  the  size  of  the  deficit  in  that  budget.  This  concern 
was  fed  by  columnists  who  trumpeted  examples  of  purported  Federal 
waste.  While  it  is  naive  to  judge  a  scientific  grant  by  its  title,  the 
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public's  concern  was  not  to  be  dismissed  without  an  investigation 
designed  to  assure  that  money  was  being  spent  for  worthwhile  purposes. 

3.  Freedom  of  Information 

A  widespread  belief  that  government  operations  ought  to  be  open  to 
public  scrutiny  to  the  greatest  extent  feasible  had  developed  during  the 
early  1970's.  National  events  showed  this  belief  to  be  well  founded,  and 
reinforced  it.  The  Freedom  of  Information  Act  and  the  Federal 
Advisory  Committee  Act  sprang  in  large  part  from  a  desire  for  openness 
in  government.  In  the  spirit  of  openness  the  National  Science  Board 
had  voted  at  its  meeting  of  June  20,  1975  to  make  changes  in  the 
Foundation's  peer  review  system  which  would  provide  fuller  informa- 
tion to  grant  applicants  and  to  the  public.  It  was  appropriate  for  the 
Subcommittee  to  examine  the  Foundation's  peer  review  application 
evaluation  procedures  to  determine  the  extent  to  which  freedom  of 
information  can  coexist  with  the  effective  evaluation  of  applications 
for  scientific  grants. 

4-  Competition  for  Foundation  Funds 

The  size  of  the  scientific  community  had  grown  rapidly  for  the  last 
decade.  Foundation  funds  for  basic  research,  expressed  in  constant 
dollars,  had  increased  only  enough  during  that  time  to  support 
continuing  Foundation  programs  and  projects  transferred  to  the 
Foundation  from  other  agencies  at  a  roughly  constant  level.  As  a 
result  Foundation  staff  members  anticipated  an  increase  in  complaints 
from  disappointed  applicants.  The  Foundation  wanted  Congress  to 
understand  its  prcedures. 

Related  events,  all  of  which  tend  to  exacerbate  the  difficulty  of 
properly  distributing  funds  for  scientific  research,  included  the  budget 
difficulties  of  colleges  and  universities  and  the  halt  in  the  growth  of 
the  nation's  scientific  faculty  at  four-year  colleges  and  universities. 

C.    HEARINGS 

1.  Focus 

The  central  question  on  which  the  hearings  focused,  and  on  which 
this  report  focuses  is,  "Once  monies  have  been  appropriated  to  the 
Foundation  and  the  division  of  those  monies  among  disciplines  and 
purposes  has  been  agreed  upon,  how  does  the  Foundation  go  about 
making  individual  awards?"  Investigation  of  the  central  question 
required  first  that  the  Subcommittee  obtain  an  improved  understand- 
ing of  how  the  National  Science  Foundation  decides  which  scientists 
and  projects  will  receive  Foundation  funds.  Second,  it  was  necessary 
to  compare  operations  at  the  Foundation  with  other  methods  for 
allocating  Federal  money  to  scientific  research  projects  in  the  private 
sector.  Investigation  was  aimed  towards  determining  whether  the 
Foundation  is  doing  a  good  job  in  this  area  of  grant  allocation.  Are  its 
decision-making  processes  theoretically  sound  and  practically 
effective? 

The  intended  focus  was  maintained  well  throughout  the  course  of  the 
hearings.  There  were  only  brief  instances  of  digression.  The  hearings, 
in  particular,  did  not  bear  on  the  issue  of  priority  setting  among 
scientific  fields  and  purposes.  This  is  an  area  of  such  magnitude  and 
importance  it  needs  and  deserves  separate,  thorough  investigation. 
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Priority  setting  relates  to  peer  review  in  at  least  two  ways,  neither  of 
which  was  addressed  during  the  course  of  the  hearings.  First,  the 
Foundation  may  assemble  a  group  of  scientists  (peers)  to  adviso  it 
explicitly  concerning  priorities.  Second,  the  accumulated  decisions  on 
individual  grants  arrived  at  through  peer  review  may  constitute  an 
implicit  setting  of  priorities. 

2.  Selection  of  Witnesses 

Two  Congressmen  asked  to  be  permitted  to  appear  as  witnesses. 
Their  requests  were  granted  as  a  matter  of  course. 

The  Subcommittee  strove  to  ensure  that  witnesses  who  appeared 
other  than  Congressmen  either  could  provide  expert  knowledge 
concerning  the  allocation  of  funds  for  scientific  giants  or  could  repre- 
sent the  viewpoint  of  some  segment  of  the  scientific  community  con- 
cerning that  allocation.  In  several  cases  witnesses  embodied  unique 
combinations  of  expertise  and  viewpoint. 

The  Subcommittee  also  wanted  to  be  sute  that  responsible  scientists 
with  negative  views  on  the  Foundation  were  fully  heard  in  order  that 
critical  comments  might  be  evaluated.  Representative  Conlan  was 
most  helpful  in  this  respect.  He  recommended  seven  witnesses  all  of 
whom  were  well-qualified  scientists,  and  several  of  whom  were  leaders 
in  their  fields  of  specialty.  In  the  interests  of  a  full  hearing,  all  seven 
were  asked  to  appear.  All  seven  were  highly  critical  of  one  aspect  or 
another  of  the  Foundation's  operation. 

In  light  of  the  purposive  selection  of  witnesses,  it  is  clear  that  the 
record  of  the  hearings  should  not  be  taken  either  as  a  cro^s-section  of 
opinion  from  the  scientific  community  or  as  a  collection  of  disinteiested 
expert  testimony,  but  rather  that  the  comments  of  each  witness  must 
be  evaluated  in  light  of  the  witness'  background  and  the  reasons  why 
the  witness  was  selected  to  testify. 

A  typology  of  witnesses  according  to  the  Subcommittee's  reasons  for 
selecting  the  witnesses  follows,  with  notes  on  exceptional  backgrounds: 

TYPOLOGY  OF  WITNESSES 

Picked  for  Competencies  and  Viewpoints 

workings  of  the  national  science  foundation 

H.  Guyford  Stever,  Director,  NSF. 

Norman  Hackerman,  Chairman,  National  Science  Board. 

Richard  Atkinson,  Deputy  Director,  NSF. 

Donald  Rice,1  member,  National  Science  Board. 

OPERATION  OF  OTHER  FEDERAL  AGENCIES 

John  Sherman,  former  acting  director,  National  Institutes  of  Health. 
William  Raney,  Chief  Scientist,  Office  of  Naval  Research. 

ACADEMIC  INSTITUTIONS 

Universities:  Charles  Kidd,2  Executive  Secretary,  Association  of  American 
Universities. 

Small  Colleges :  James  Powell,  Provost,  Oberlin  College. 


i  President,  RAND  Corporation. 

>  Author  of  a  book  on  the  conduct  of  Federal  research  grant  programs. 
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SCIENTIFIC  SOCIETIES 

William  Carey,8  Executive  Director,  American  Association  for  the  Advance- 
ment of  Science. 

ENGINEERING  VIEWPOINT 

Donald  Marlowe  4  (et  al),  Executive  Director,  American  Society  for  Engineering 
Education. 

WORKING  SCIENTIST 

Raymond  Bowers  5,  Professor,  Cornell  University. 

REVIEWER  OF  ISIS  PROPOSAL 

Philip  Morrison,  Professor,  Massachusetts  Institute  of  Technology. 

Congressmen  Who  Requested  To  Testify 
John  Conlan,  of  Arizona.  Robert  Bauman,  of  Maryland. 

Witnesses  Recommended  by  Mr.  Conlan 
materials  research  viewpoint 

Doris  Wilsdorf,6  professor,  University  of  Virginia. 

Heinz  Wilsdorf,  department  chairman,  University  of  Virginia. 

Rustum  Roy,6  department  chairman.  Pennsylvania  State  University. 

biomedicine  innovation  viewpoint 

Gilbert  Ling,  department  director,  Pennsylvania  Hospital. 
Carlton  Hazlewood,  associate  professor,  Baylor  College  of  Medicine. 
Freeman  Cope,  biophysicist,  U.S.  Naval  Air  Development  Center  Biochemical 
Laboratory. 

CURRICULUM    DEVELOPMENT    PROJECT    MONITORING    VIEWPOINT 

Elmer  Seevers,  science  curriculum  coordinator,  Biomedical  Interdisciplinary 
Curriculum  Project. 

3.  Inquiry 

The  Subcommittee  took  pains  to  conduct  a  thorough  inquiry  into 
National  Science  Foundation  peer  review.  Witnesses  were  questioned 
in  depth  routinely  by  Chairman  Symington  and  by  the  ranking  minor- 
ity member  of  the  Subcommittee,  Charles  Mosher  of  Ohio.  Other 
Subcommittee  members  participated  to  varying  extents  in  the  ques- 
tioning. As  a  result  of  the  acceptance  of  all  recommended  witne 
and  the  detailed  questioning  of  all  witnesses,  the  hearings  which  were 
originally  scheduled  to  last  twelve  hours  spread  over  six  days  (July  22. 
23,  24,  29,  30,  and  31,  1975)  in  fact  required  a  total  of  over  21  hours 
on  those  same  days. 

Following  the  hearings  the  Subcommittee,  in  addition  to  its  own 
staff,  employed  the  Congressional  Research  Service  for  a  disinterested 
interpretation  and  evaluation  of  particularly  technical  testimony  and 
asked  the  General  Accounting  Office  to  check  the  use  by  the  Founda- 
tion of  a  particular  collection  of  verbatim  peer  reviews. 

For  the  hearings  record,  in  addition  to  the  spoken  and  written 
testimonies  with  their  attachments,  the  Subcommittee  solicited. 
received  and  inserted  some  two  dozen  further  submissions  and  cata- 
logued a  like  number  of  letters  concerning  peer  review  that  it  had 
received.   A  copy   of   the   table   of  contents   of   the    record    follows: 


1  Formerly  a  high  official  in  the  (now)  Office  of  Management  and  Budget. 

♦  Dr.  Marlowe  (etal)  asked  to  testify. 

»  Also  Director  of  the  Program  on  Science,  Technology  and  Society  at  Cornell  University. 

•  Also  member,  NSF  Advisory  Committee  on  Metallurgy  and  Materials. 
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III.  Peer  Review  Systems 

A.    GENERAL 

Most  broadly  speaking  a  "peer  review  system"  is  any  method  of 
evaluating  a  specialized  creation — such  as  a  proposal  to  perform 
scientific  research — which  involves  having  a  group  of  people  knowl- 
edgeable in  the  area  of  specialization  evaluate  the  creation.  The 
experts  are  called  "peers,"  the  term  loosely  deriving  from  the  relation- 
ship between  the  proposer  and  the  evaluators.  In  science  the  chief  uses 
of  peer  review  are  to  help  determine  whether  a  proposed  piece  of  work 
or  study  should  be  funded  or  to  help  determine  whether  an  existing 
scientific  paper  should  be  published.  Peer  review  is  used  for  these  pur- 
poses because  it  is  believed  to  be  the  best  practical  method  of  identi- 
fying excellence. 

The  Subcommittee's  interest  in  peer  review  stems,  of  course,  from 
its  use  by  the  National  Science  Foundation  to  evaluate  proposals. 
Even  in  this  more  restricted  sense,  "peer  review"  is  very  broadly 
defined.  Two  particular  pitfalls  of  definition  should  be  borne  in  mind 
in  any  consideration  of  peer  review.  First,  the  label  "peer  review 
system"  does  not  reveal  who  is  actually  making  the  decision  on  whether 
or  not  to  fund  a  given  proposal;  it  may  be  the  group  of  peers,  or  it 
may  be  the  person  or  organization  controlling  the  money.  Second,  the 
"peers"  involved  in  a  peer  review  are  often  not  the  equals  of  the 
proposer.  For  example,  proposals  from  postdoctoral  students  may  be 
evaluated  by  full  professors. 

B.    ELEMENTS 

Following  is  a  list  of  the  most  important  elements  used  by  Federal 
agencies  in  systems  for  deciding  what  extramural  scientific  projects 
should  be  supported.  An  agency  will  have  established  its  programmatic 
priorities  (including  how  much  money  should  be  spent  on  each 
program)  by  other  means  before  reaching  the  point  at  which  individual 
projects  are  considered.  The  list  omits  accounting  and  business 
reviews,  which  do  take  place.  Consideration  of  a  given  proposal 
requiies  some,  but  rarely  all,  of  the  elements.  Particular  but  diffeiing 
patterns  of  use  occur  in  diffeiing  agencies  and  programs. 

1.  Staff  Judgement 

In  most  systems  agency  staff  judgement  is  a  key  element.  At  one 
extreme  of  reliance  on  staff  in  a  peer  review  system,  the  staff  may 
determine  how  a  proposal  will  be  reviewed,  select  the  reviewers,  use 
the  reviewers  only  as  advisors,  and  make  the  decision  on  funding.  At 
the  opposite  extreme  the  staff  may  perfoim  only  clerical  functions: 
mailing  proposals  to  reviewers  on  a  predetermined  list,  compiling 
evaluation  scores,  etc. 

Staff  judgement  may  be  exercised  by  individuals  or  by  panels  of 
staff  members.  Grant  review  boards  at  the  Foundation  are  examples 
of  staff  panels.  Frequently,  judgements  made  at  the  initial  level  by 
any  procedure  receive  administrative  review  by  an  individual  in  the 
next  hierarchical  level  (and  possibly  succeeding  levels). 

(13) 
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2.  Individual  Peer  Review 

Individual  peer  review  occurs  when  an  expert  outside  the  agency 
staff  evaluates  a  proposal  without  consulting  other  evaluators. 
Typically,  copies  of  a  proposal  will  be  mailed  to  three  to  ten  peers  for 
individual  peer  review  and  returned  to  the  agency. 

3.  Panel  Peer  Review 

This  may  take  several  forms.  Most  commonly  a  group  of  expeits 
meets  together  to  consider  the  merits  of  a  set  of  proposals.  The  panel 
may  be  a  standing  group  (generally  with  rotating  membership)  or  it 
may  be  an  ad  hoc  group  with  a  brief  existence. 

4-  Site  Visit 

Site  visits  may  be  used  in  a  variety  of  situations  and  may  take  a 
variety  of  forms.  The  site  visit  team  may  comprise  a  single  staff 
person,  several  staff  people,  or  a  combination  of  staff  and  outside 
experts.  A  site  visit  may  be  necessary  when  the  quality  of  scientific 
facilities  at  a  site  is  a  determinant  of  the  likelihood  of  success  of 
proposed  research. 

5.  Policy  Board 

In  some  cases  the  question  whether  a  given  proposal  fits  into  an 
agency's  program  is  particularly  crucial,  or  other  policy  difficulties 
may  be  raised  by  a  proposal.  In  such  instances  the  proposal  may  be 
referred  to  a  policy  board.  An  example  is  the  National  Science  Board 
at  the  Foundation.  Policy  boards  are  usually  standing  groups  with 
rotating  membership. 

6.  Consultation  With  Other  Funding  Agencies 

Consultation  is  frequently  necessary.  It  is  engendered  by  proposals 
requesting  partial  funding  from  various  agencies,  by  requests  for  use 
of  government  facilities,  by  overlapping  projects,  etc. 

C.   NATIONAL  SCIENCE  FOUNDATION  PEER  REVIEW 

The  Subcommittee's  focus  in  this  report  is  on  the  National  Science 
Foundation's  methods  for  making  individual  awards.  Either  individual 
or  peer  panel  review  is  used  by  the  Foundation  in  the  evaluation  of 
the  vast  bulk  of  formal  requests  for  funds.  Hence,  the  Foundation's 
evaluation  methods  are  generally  called  peer  review  systems.  These 
methods  were  described  in  "Peer  Review  and  Proposal  Evaluation," 
a  Foundation  staff  study  dated  June  1975,  which  is  included  in  the 
peer  review  hearings  record.  Particular  aspects  of  certain  of  the 
Foundation's  procedures  were  described  in  greater  detail  during  the 
questioning  periods  at  the  hearings. 

In  Fiscal  Year  1974  about  21,000  actions  were  taken  on  project 
proposals  submitted  to  the  Foundation.  Very  nearly  half  the  proposals 
were  funded.  Expenditures  of  $600  million  were  made  for  programs  in 
that  year  by  the  Foundation.  On  the  average  each  proposal  received 
around  6  reviews.  Roughly  speaking,  half  the  proposals  received  in- 
dividual peer  review  only,  a  quarter  received  peer  panel  review  only, 
and  a  quarter  received  both.  The  number  of  different  reviewers  was 
close  to  30,000  and  those  reviewers  submitted  approximately  120,000 
reviews. 

The  Foundation  is  unique  among  Federal  agencies  in  its  mission 
"to  initiate  and  support  basic  scientific  research  and  programs  to 
strengthen    scientific    research    potential  .  .  ."    (without   regard    to 
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applicability  of  research  to  immediate  problems).  The  procedures  of 
the  Foundation  for  making  awards  have  been  largely  shaped  by  its 
pursuit  of  this  mission.  Other  missions  of  the  Foundation  are  to  support 
science  education  and  to  support  immediately  applicable  research  not 
adequately  supported  by  other  agencies.  Decision-making  procedures 
for  awards  stemming  from  these  missions  are  different. 

A  summary  of  the  Foundation's  procedures  for  making  individual 
grants  follows.  The  summary  describes  only  the  mainstream  of  Foun- 
dation procedures.  It  is  included  here  as  a  point  of  departure  from 
which  to  interpret  the  issues  and  analyze  the  testimony.  A  deeper 
discussion  of  many  aspects  of  the  procedures  is  included  in  the 
"Analysis  of  Testimony"  chapter  of  this  report. 

1.  General 

Given  that  the  Foundation's  priorities  among  programs  and  pur- 
poses have  been  established  previously,  the  Foundation's  methods  for 
deciding  who  will  receive  an  individual  award  begin  at  the  point  where 
it  is  decided  what  form  an  award  will  take.  In  the  vast  majority  of 
cases,  the  award  is  a  grant  to  or  contract  with  an  institution  ('which 
is  financially  responsible)  to  be  used  in  the  support  by  the  institution 
of  a  project  under  the  direction  of  a  principal  investigator  (who  is 
responsible  for  the  work).  A  second  type  of  award  is  for  education 
(the  Foundation's  Graduate  Fellowships  are  the  foremost  example), 
in  which  there  is  an  institution  and  a  person,  but  no  project.  A 
third  type  of  award  is  a  grant  to  an  institution  to  improve  the  insti- 
tution. The  Foundation  does  not  normally  make:  (1)  awards  to  indi- 
viduals in  the  absence  of  an  institution  to  >pon-or  the  individual,  or 
(2)  research  grants  to  institutions  for  a  particular  seienti-t,  not 
specifying  that  work  in  a  particular  field  be  done. 

The  Foundation  acts  on  applications  for  funds  in  deciding  who  will 
get  awards.  These  applications  may  be  received  unsolicited  or  may  have 
been  solicited  by  the  Foundation  from  the  scientific  and  technical 
community  either  in  a  request  for  proposals  for  undertakings  in  a 
given  broad  field  (a  program  solicitation)  or  in  a  request  for  proposals 
to  perform  a  project  that  has  been  defined  by  the  Foundation. 

Nearly  all  Foundation  review  procedures  involve  heavy  participa- 
tion by  Foundation  staff  members  at  the  point  where  a  decision  i^ 
reached  on  an  application  for  funds.  AYhen  comparatively  small 
amounts  are  involved  in  an  individual  award — as  is  the  case  for  foreign 
travel  awards — staff  judgment  alone  may  be  deemed  sufficient  in 
arriving  at  a  decision. 

2.  Basic  Research 

Over  half  the  funds  expended  by  the  Foundation  go  for  basic 
scientific  research  (over  $300  million  annually).  Most  basic  research 
awards  are  for  amounts  in  the  range  of  810,000  to  $100,000.  In  addition 
to  staff  judgment,  all  basic  research  support  programs  use  individual 
peer  review  for  evaluating  applications.  The  use  of  peer  panels  i- 
mixed;  fewer  than  half  the  Foundation's  basic  research  programs  use 
peer  panels.  Other  review  elements  are  employed  in  varying  degrees 
by  the  various  programs. 

The  following  flow  chart  and  list  of  processes  >how  the  procedures 
used  in  the  evaluation  of  an  application  by  the  former  Directorate 
for  research.  The  procedures  are  still  basically  the  same  although  the 
Foundation  has  reorganized. 
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Flow  Chart  of  the  Movement  of  a  Research  Proposal  Through  the  Foundation 
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Footnotes  to  Chart 

1.  Principal  Investigator  (P.T.)  conceives  research  plan. 

2.  P.T.  may  hold  preliminary  discussions  with  Program  Director  (P.D.)  and/or  submit  preliminary 
proposal  for  evaluation. 

3.  Final  proposal  is  prepared  by  P.I.,  approved  by  Institution  and  sent  to  NSF. 

4.  Central  Processing  assigns  proposal  to  a  Division. 

5.  Division  Director  assigns  proposal  to  a  Section. 

6.  Section  Head  assigns  proposal  to  a  Program. 

7a.  Program  Director  chooses  reviewers  and  sends  out  proposal  for  independent  mail  review. 

7b.  Program  Director  may  choose  panel  members  and  send  them  copies  of  proposal  in  preparation  for 
panel  meeting. 

7c.  Program  Director  may  discuss  proposal  with  another  agency. 

7d.  Program  Director  may  discuss  proposal  with  other  components  of  NSF. 

7e.  Program  Director  may  make  site  visit  (orsite  visiting  team  may  be  appointed  and  report  back  to  P.D.). 

8a.  Independent  mail  reviewers  evaluate  proposal  and  return  signed,  written  reviews. 

8b.  Panel  members  discuss  proposal  and  indicate  rating. 

9a.  Program  Director  may  decline  proposal  but  suggest  some  major  modification  that  would  make  it 
more  acceptable  or  may  suggest  that  a  new  proposal  may  be  written. 

9b.  Program  Director  may  decide  to  recommend  funding  and  negotiates  revised  budget  with  P.I. 

10.  P.D.  recommends  funding  of  revised  amount. 

11.  Section  Head  reviews  recommendation,  approves  and  transmits,  or  rejects. 

12.  Division  Director  reviews  recommendation,  approves  and  transmits,  or  rejects. 

13.  Assistant  Director  for  Research  may  review  recommendation. 

14.  If  grant  is  large  enough,  National  Science  Board  must  review. 

15.  Grant  is  made  to  Institution,  which  disburses  funds  to  P.I.'s  project. 
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8.  Research  Applications 

In  recent  years  annual  expenditures  by  the  Foundation  for  applied 
research  have  been  around  $70  million.  Most  applied  research  grant 
amounts  are  in  the  range  $40,000  to  $400,000,  thus  averaging  two  or 
three  times  the  size  of  basic  research  grants. 

The  Research  Applications  Directorate  of  the  Foundation  seeks  to 
fund  projects  that  will  result  in  benefits  soon  after  completion.  Usually, 
the  Research  Applications  Directorate  has  a  particular  benefit  in  mind 
and  applications  are  solicited  for  projects  generally  designed  to  achieve 
a  known  goal  (a  program  solicitation).  In  same  cases,  the  project 
necessary  to  achieve  a  benefit  can  be  described  beforehand  and  the 
Foundation  will  issue  a  "request  for  proposal"  (RFP).  The  Foundation 
must  also  deal  with  unsolicited  proposals  for  applied  research  which 
do  not  fit  its  predesigned  programs.  Thus,  three  different  types  of 
proposal  solicitation  and  evaluation  are  used  by  the  Research  Applica- 
tions Directorate. 

1.  Proposals  submitted  under  program  solicitations  are  screened  by 
the  staff.  Those  judged  by  the  staff  to  be  above  a  minimum  quality 
level  generally  receive  peer  panel  review.  The  peer  panel  commonly 
considers  a  batch  of  proposals  addressed  to  a  given  program  solicita- 
tion and  makes  the  primary  decision.  The  decision  is  reviewed  at  higher 
staff  levels. 

2.  The  evaluation  of  proposals  submitted  in  response  to  an  RFP  is 
closely  governed  by  Federal  regulations.  Primary  decision-making  on 
such  proposals  in  the  Research  Applications  Directorate  lies  with  a 
panel  usually  composed  of  scientifically  or  technically  qualified  Federal 
employees. 

3.  An  unsolicited  proposal  is  generally  discussed  with  the  Founda- 
tion's program  manager  before  being  submitted;  this  constitutes  a 
pre-review.  A  formally  submitted  unsolicited  proposal  will  receive 
individual  peer  review.  The  staff  makes  the  primary  decision,  which  is 
reviewed  at  higher  staff  levels. 

4-  Education 

Two  quite  dissimilar  types  of  application  are  handled  in  the  Educa- 
tion Directorate:  fellowship  applications  and  project  support  ap- 
plications. 

About  $15  million  is  spent  annually  for  fellowships  at  an  average 
expenditure  of  somewhat  less  than  $10,000  each.  Fellowship  applica- 
tions are  reviewed  and  evaluated  by  ad  hoc  panels.  The  primary 
decision  lies  with  the  panels  themselves. 

The  Foundation's  expenditures  for  other  science  education  activities 
are  of  the  order  of  $50  million  annually.  Project  support  applications 
are  received  in  several  different  Education  Directorate  programs. 
About  three-fourths  of  such  applications  are  evaluated  through  ad  hoc 
peer  panel  review.  Standing  peer  panels  are  not  used.  Approximately 
a  fifth  of  the  Directorate's  proposals  receive  individual  peer  review. 

5.  Other 

National  and  Special  Research  programs  of  the  Foundation  have 
required  around  $90  million  annually  (the  Ocean  Sediment  Coring 
Program  excluded) .  The  Foundation's  methods  of  reviewing  proposals 
in  these  programs  vary  widely.  Individual  peer  review  is  the  most 
common  adjunct  to  staff  review.  The  programs  usually  last  many 
years.  This  allows  the  evaluation  of  new  proposals  to  lean  on  reports  of 
work  already  performed. 
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National  Research  Centers  (plus  the  Ocean  Sediment  Coring 
Program)  expend  about  $60  million  annually.  The  Centers  are  each 
funded  by  the  Foundation  through  relatively  few,  but  large,  contracts. 
In  its  decision-making  with  respect  to  the  Centers,  the  Foundation 
appears  to  determine  how  much  it  wishes  to  spend  on  the  Center-  as 
opposed  to  other  programs,  not  as  opposed  to  other  projects.  Hence, 
decision-making  on  Center  funding  falls  outside  the  scope  of  this 
report. 

D.    OTHER 

Many  Federal  agencies  make  awards  for  basic  scientific  research. 
Representatives  of  two  such  agencies,  the  National  Institutes  of 
Health  and  the  Office  of  Naval  Research,  appeared  at  the  hearings 
and  presented  the  methods  used  by  those  agencies  to  decide  to  whom 
awards  will  be  given,  as  a  contrast  to  the  procedures  of  the  National 
Science  Foundation.  The  Subcommittee  used  this  testimony  to  under- 
stand what  differences  in  mission  or  environment  would  explain  the 
differences  in  procedures  of  various  agencies. 

1.  National  Institutes  oj  Health 

The  mission  of  the  National  Institutes  of  Health  (NIH)  is  the  conduct 
and  support  of  research  to  enhance  the  understanding  of  health  and 
disease  so  as  to  improve  the  health  and  well  being  of  the  people  of  the 
nation.  NIH  both  operates  its  own  laboratories,  clinics,  etc.  and 
supports  the  performance  of  scientific  projects  by  other  institutions, 
chiefly  universities.  NIH  has  a  program  of  extramural  contracts  and 
a  much  larger  program  of  extramural  research  grants.  The  Subcommit- 
tee focused  its  attention  on  the  extramural  research  grant  program. 

In  Fiscal  Year  1974  NIH  disbursed  $569  million  for  about  11.000 
extramural  research  grants  to  nearly  1,100  institutions.  In  that  year 
the  NIH  research  grant  program  considered  10,000  new  applications; 
slightly  fewer  than  half  were  funded.  An  additional  6,600  applications 
for  continuing  funding  for  previously  approved  projects  were  approved 
as  a  matter  of  course. 

NIH  procedures  for  deciding  who  will  receive  research  grants  are 
more  regular  than  those  employed  by  the  Foundation.  The  procedures 
were  described  in  the  testimony  of  Dr.  John  Sherman,  former  acting 
director  of  NIH,  which  appears  in  the  peer  review  hearings  record. 

Peer  panel  reviews  by  standing  "study  sections"  with  rotating 
membership  are  employed  as  the  key  decision-making  element  in  the 
NIH  system.  In  addition  to  peer  panel  review  by  study  section. 
applications  recommended  for  approval  by  the  study  sections  receive 
policy  board  review  by  an  advisory  council  (one  council  is  mandated 
by  law  for  each  of  the  eleven  NIH  institutes).  This  policy  board  re- 
view changes  the  status  of  a  proposal  in  comparatively  few  cases. 

Currently  NIH  maintains  about  50  study  sections,  each  of  which 
has  16  or  so  members  (the  total  number  of  peer  reviewers  used  annually 
by  NIH  to  help  judge  research  grant  applications  is  thus  around  1 ,000) . 
NIH  does  not  find  it  necessary  routinely  to  publish  program  solicita- 
tions in  order  to  obtain  adequate  grant  applications.  The  field  of  science 
covered  by  the  interests  of  the  eleven  institutes  is  very  broad,  and 
the  community  of  applicants  is  able  to  address  those  interests  effec- 
tively in  its  unsolicited  applications.  When  an  application  is  received 
a  central  office  assigns  it  to  both  a  study  section  and  an  institute. 
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Most  study  sections  are  apt  to  review  proposals  for  several  institutes 
although  some  relate  primarily  to  a  single  institute.  Each  study  section 
is  served  by  a  full-time  NIH  employee  called  an  executive  secretary 
(corresponding  roughly  to  a  Foundation  program  director).  Study  sec- 
tions normally  meet  three  times  annually  and  consider  a  batch  of 
applications  at  each  meeting.  The  executive  secretary  is  responsible  to 
see  that  sufficient  information  is  available  about  each  application.  A 
site  visit  or  individual  peer  reviews  ma}'  be  employed  in  infrequent 
cases,  the  results  of  which  are  provided  to  the  study  section  to  aid  in 
the  evaluation  of  an  application. 

Each  application  considered  is  recommended  for  approval  or  dis- 
approval or  is  deferred  for  further  collection  of  information  at  the  study 
section  meeting.  Approved  applications  receive  a  numerical  priority 
rating  based  primarily  on  scientific  merit,  and  a  summary  statement  is 
prepared  based  on  the  written  reviews  and  comments  of  study  section 
members.  The  primary  determinants  of  whether  a  given  proposal  will 
be  funded  are  its  priority  score  and  the  amount  of  funds  available  in 
the  institute  of  NIH  to  which  the  proposal  has  been  assigned.  The 
priority  rankings  of  the  study  sections  are  used  to  prepare  a  priority- 
ordered  list  of  proposals  (which  may  involve  projects  reviewed  by 
several  study  sections)  for  presentation  to  each  institute  advisory 
council.  The  council  may  change  the  list  to  conform  with  institute 
programmatic  policies.  As  noted  earlier,  changes  are  not  usually 
extensive.  Grants  are  made  in  each  institute  according  to  the  priorities 
agreed  upon  by  the  advisory  council  until  the  institute's  research  fund- 
are  exhausted,  but  by  law,  no  grant  may  be  funded  unless  recom- 
mended by  the  council.  Currently,  from  13  to  60  percent  of  approved 
proposals  are  funded,  depending  upon  the  institute. 

2.  Office  oj  Naval  Research 

The  mission  of  the  Office  of  Naval  Research  (ONR)  as  described 
in  its  statutory  authority  and  executed  by  ONE,  is  "to  plan,  foster 
and  encourage  scientific  research  in  recognition  of  its  paramount 
importance  as  related  to  the  maintenance  of  future  naval  power,  and 
the  preservation  of  national  security."  Like  NIH,  the  Navy  has 
extensive  in-house  laboiatories  but  supports  extramural  research  as 
well  (the  similarity  does  not  extend  much  farther).  ONR  supports 
extramural  research  for  the  good  of  the  Navy  rather  than  for  the 
general  national  good.  Nevertheless,  much  work  ONR  supports  is 
basic  research. 

ONR  distributes  roughly  $60  million  annually  for  scientific  research 
awards.  Contracts  rather  than  grants  are  used  as  the  award  instru- 
ment, but  the  type  of  award  seems  not  to  have  much  interrelationship 
with  the  decision-making  process  that  leads  to  the  award. 

Methods  used  by  ONR  to  decide  who  will  receive  awards  were 
de>cribed  at  the  peer  review  hearings  by  William  Raney,  Chief  Sci- 
entist at  ONR.  The  key  element  in  the  ONR  decision-making  process 
is  the  ONR  staff.  ONR  "scientific  officers,"  the  counterparts  to  the 
Foundation's  program  officers,  are  expected  to  maintain  a  strong 
orientation  towards  serving  the  needs  of  the  Navy.  ONR  establishes 
programs  designed  to  meet  perceived  Navy  needs  and  publicizes  its 
programs  for  the  benefit  of  potential  applicants  in  Opportunities  for 
Naval  Research.  ONR  interests  are  also  made  known  through  sympo- 
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sia,  workshops,  etc.,  and  the  contacts  of  ONE,  scientific  officers  with 
the  research  community. 

The  scientific  officer  is  also  dominant  in  the  proposal  review  part  of 
ONR's  decision-making  process.  The  scientific  officer  determines  how 
each  proposal  will  be  reviewed;  there  is  no  ONR-wide  approved 
method.  At  the  discretion  of  the  officer,  individual  peer  review  or 
panel  peer  review  may  be  employed — and  have  been  employed  in 
many  instances.  Alternatively  the  scientific  officer  may  make  a  deci- 
sion himself  or  herself  or  call  on  other  Federal  agencies  for  advice,  or 
undertake  different  types  of  review,  as  deemed  appropriate  to  the 
particular  situation. 

8.  Scientific  Articles 

Most  professional  journals  and  many  scientific  and  technical  maga- 
zines use  a  peer  review  procedure  to  select  papers  or  articles  for 
publication  from  among  unsolicited  submissions  of  the  interested 
community.  The  journal  editor  mails  copies  of  a  submitted  paper  to 
one  or  several  reviewers  who  read  the  paper,  edit  and/or  comment  on 
it,  and  return  it.  The  editor  decides  whether  to  print  the  paper  largely 
on  the  basis  of  the  reviewers'  advice.  Reviewers'  comments  are 
normally  made  available  verbatim  to  the  author  with  the  reviewers' 
names  deleted. 

Several  witnesses  referred  to  their  experience  as  editors  with  peer 
review.  There  are  essential  differences  between  the  review  of  a  paper 
and  the  review  of  a  proposal,  however,  which  limit  the  validity  of 
comparing  the  two  uses  of  peer  review.  Foremost  among  the  differ- 
ences is  that  a  paper  is  a  finished  piece  of  work  and  may  be  evaluated 
with  much  less  regard  to  the  scientific  ability  of  its  author  than  a 
proposal.  In  some  fields  papers  are  reviewed  with  the  author's  name 
deleted;  a  proposal  evaluation,  on  the  other  hand,  always  requires  the 
reviewer  to  judge  the  applicant. 


IV.  Issues 

Peer  review  provides  a  wealth  of  issues.  The  issues,  moreover,  are 
not  discrete  but  interact  strongly  with  one  another.  They  may  be  cast 
in  four  constellations  and  a  group  of  miscellany.  The  four  broad  areas 
are: 

Does  the  National  Science  Foundation  support  high-quality 
research? 

What  do  all  the  people  involved  in  the  Foundation's  decision- 
making processes  do?  Are  they  doing  the  right  things,  and  are 
they  doing  them  right? 

How  open  and  fair  is  the  Foundation's  S3rstem  of  making 
awards? 

What  distribution  of  money  results  from  the  Foundation's 
decision-making  processes?  Is  it  desirable? 

A.    SUPPORT    OF    HIGH-QUALITY    RESEARCH 

The  most  important  single  consideration  is  whether  the  National 
Science  Foundation's  awards  lead  to  the  advance  of  science  in  the  most 
effective  possible  way.  Regardless  of  any  other  factors,  if  the  Founda- 
tion is  not  supporting  good  research  and  education,  it  is  not  fulfilling 
the  charge  Congress  has  given  it.  How  do  the  Foundation's  decision- 
making procedures  relate  to  this  charge? 

In  this  issue  it  is  useful  to  distinguish  between  research  that  is  good 
but  orthodox  and  research  that  is  innovative  (and  good). 

1.  Generally  Good  Research 

Do  the  Foundation's  decision-making  procedures  lead  to  the  support 
of  projects  at  the  upper  end  of  the  quality  scale?  Would  other  pro- 
cedures select  better  projects?  In  an  absolute  sense  how  high  is  the 
average  quality  of  the  funded  projects? 

2.  Innovative  Research 

The  most  significant  advances  in  science,  those  associated  with 
names  like  Darwin,  Einstein,  Freud,  Galileo,  and  Pasteur,  challenge 
the  mainstream  of  scientific  theory  or  religious  or  philosophic  belief. 
The  Foundation's  procedures  for  making  awards  should  not  prevent 
the  Foundation  from  supporting  innovative  theories.  Large  innova- 
tions provide  the  girders  that  structure  science.  Research  that  is  good 
but  not  innovative  is  the  mortar  and  bricks  that  fill  in  the  gaps.  With- 
out innovations,  good  research  runs  out  of  gaps  to  fill,  and  science  is 
more  limited  than  nature  would  allow. 

Does  the  Foundation  adequately  support  innovative  research? 

B.    MANAGING    PEER    REVIEW    AT    THE    FOUNDATION 

1.  Management  Policy 

How  are  the  basic  policies  for  the  Foundation's  decision-making 
procedures  established?  Who  should  be  establishing  the  policies: 
Congress,  the  National  Science  Board,  or  the  Foundation  staff?  Are 
the  present  policies  adequate? 

(21) 
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2.  NSFStaf 

What  role  does  the  Foundation's  staff  play  in  the  decision-making 
process?  Is  it  a  practicable  role?  Do  the  Foundation's  staff  members 
perform  their  roles  in  an  honest,  competent  way,  in  accordance  with 
Foundation  policies? 

3.  Reviewers 

What  role  do  reviewers  play  in  the  decision-making  process?  Is  it  a 
practicable  role?  Are  they  conscientious,  honest,  and  competent? 

The  most  common  objection  to  the  use  of  peer  review  is  that  it  is 
subject  to  "backrubbing"  or  "old  boys  clubs,"  in  which  mutual 
friends  unduly  praise  each  other's  proposals.  To  what  extent  does 
this  happen? 

Are  reviewers  selected  fairly  and  ration  ally?  What  is  the  distribu- 
tion of  reviewing  among  potential  reviewers?  Is  this  an  appropriate 
distribution? 

Are  reviewers  utilized  in  the  most  effective  and  fair  manner?  In 
particular,  should  the  mix  of  individual  review  and  panel  peer  review 
have  its  proportions  changed?  Suggestions  have  been  made  from 
several  quarters  that  reliance  on  panel  peer  review  should  be  increased. 

4-  Division  of  Responsibility 

In  most  decision-making  systems  for  scientific  awards  it  is  possible 
to  identify  a  point  or  interval  in  the  flow  of  events  where  the  decision 
is  actually  made.  This  key  point  is  likely  to  be  preceded  by  advisory 
activities  and  followed  by  checking  activities,  but  rarely  is  the  decision- 
making smeared  out  along  the  whole  process. 

Should  this  key  point  be  moved  closer  to  the  Foundation  staff  or 
closer  to  the  reviewers?  Should  the  key  point  be  spread  out  more 
along  the  process? 

5.  Policing  the  System 

Is  the  Foundation's  decision-making  system  for  individual  awards 
working  as  it  is  supposed  to  work?  What  ways  are  available  to  deter- 
mine the  answer  to  that  question?  What  is  the  role  of  trust  vs.  policing 
in  keeping  the  system  working? 

6.  Congressional  Oversight 

During  the  authorization  of  the  Fiscal  Year  1976  National  Science 
Foundation  budget,  the  House  of  Representatives  voted  at  one 
point  to  require  Congressional  review  of  individual  research  awards. 
The  Senate  objected  to  this  requirement,  and  it  was  eliminated  from 
the  Foundation's  authorization  bill  in  the  conference  agreement  on  the 
bill. 

The  Subcommittee  addressed  the  question  once  again:  should 
Congressional  review  be  required  before  individual  National  Science 
Foundation  awards  are  made? 

C.    INFORMATION    EXCHANGE    AND    APPEALS 

1 .  Degree  of  Openness 

Independent  of  its  effect  on  the  grant  application  decision-making 
process,  to  what  extent  is  openness  of  the  process  desirable?  How  far 
can  openness  coexist  with  the  effective  evaluation  of  applications? 
In  particular,  should  the  applicant  be  provided  with  verbatim  re- 
views of  the  application  or  with  the  reviewers'  names? 
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2.  Feedback  to  Applicants 

What  information  should  be  provided  to  an  applicant  concerning 
the  process  that  led  to  the  Foundation's  decision  on  his  or  her 
application? 

8.  Appeals 

Should  the  Foundation  have  a  more  formal  means  of  reconsidering 
its  decision  on  an  application  than  it  has  now? 

D.    DISTRIBUTION    OF    FUNDS 

The  law  regarding:  NSF  provides  that  "it  shall  be  one  of  the  objec- 
tives of  the  Foundation  to  strengthen  research  and  education  in  the 
sciences,  including  independent  research  by  individuals,  throughout 
the  United  States,  and  to  avoid  undue  concentration  of  such  research 
and  education." 

What,  in  fact,  is  the  distribution  of  the  Foundation's  funds?  Are 
there  "undue  concentrations"?  To  what  extent  is  the  distribution  of 
funds  a  result  of  peer  review  procedures?  If  there  are  imbalances,  how 
should  they  be  corrected? 

1.  Geographical 

What  is  the  fair  way  to  measure  the  concentration  of  research  and 
education  on  a  geographical  basis?  Current  funds  are  commonly 
accepted  as  a  surrogate  for  "research  and  education,"  and  the  State  is 
commonly  accepted  as  a  geographic  unit,  but  to  what  demographic 
feature  of  the  State,  if  any,  should  the  distribution  of  funds  for  that 
State  be  proportional?  Population,  number  of  scientists  and  engineers, 
and  many  other  measures  have  been  suggested.  If  some  kind  of  pro- 
portionality is  judged  desirable,  how  far  from  exact  must  the  propor- 
tions be  to  become  "undue?" 

Is  the  law  regarding  "undue  concentration"  still  the  will  of  Congress? 

2.  Institutional 

Are  proposals  from  the  faculty  of  undergraduate  institutions  fairly 
considered  relative  to  those  from  the  faculty  of  graduate  institutions? 
On  the  graduate  level,  does  the  reputation  of  the  proposer's  institution 
have  too  great  an  influence  on  the  decision  to  fund  a  proposal? 

3.  Young  Scientists 

It  is  frequently  said  that  scientific  research  is  a  young  man's  game. 
Are  young  scientists  adequately  supported  by  the  Foundation? 

E.    OTHER 

Several  issues  were  discussed  at  the  peer  review  hearings  which  were 
tangentially  related  or  unrelated  to  peer  review. 

1.  National  Science  Foundation  Programs 

Three  specific  Foundation  programs  were  addressed  by  different 
sets  of  witnesses:  materials  research,  engineering,  and  biochemistry. 

2.  Overlap  oj  Research 

To  what  extent  does  the  Foundation  support  research  in  areas  of 
interest  to  other  agencies  which  also  support  research?  Is  this  state 
of  affairs  desirable? 


V.  Analysis  of  Testimony 

A.    SUPPORT    OF    HIGH-QUALITY    RESEARCH 

Witnesses  agreed  overwhelmingly  that  some  form  of  peer  review 
should  continue  to  be  used  to  assist  in  the  allocation  of  Federal  funds 
for  scientific  research.  Not  a  single  witness  suggested  that  peer  review 
be  abandoned,  although  several  witnesses  proposed  changes  in  the 
decision-making  processes  of  the  Foundation — some  minor  and  some 
major  in  their  potential  effects. 

1 .  Generally  Good  Research 

The  basic  rationale  for  the  use  of  peer  review  in  decision-making 
for  scientific  awards  is  that  the  experts  in  a  field  of  science  are  better 
able  to  judge  several  important  factors  than  anyone  else.  These  factors 
include: 

Past  performance  of  the  proposer. 

Design  of  the  proposed  work. 

Importance  of  proposed  work  to  the  scientific  field. 

In  particular,  most  witnesses  agreed  that  the  NSF  staff  could  not 
perform  as  well  without  the  benefit  of  peer  review  as  with  it.  Indeed, 
several  witnesses  stated  that  the  potential  for  staff  abuse  is  greater  in 
systems  lacking  peer  review. 

There  was  not  a  great  deal  of  testimony  addressing  directly  the 
issue  of  the  quality  of  projects  supported  by  the  Foundation.  It  may 
be  fairly  inferred  from  the  testimony,  however,  that  witnesses  agreed 
almost  unanimously  that  appropriate  peer  review  procedures  would 
tend  to  select  for  funding  those  projects  at  the  upper  end  of  the 
quality  scale,  however  high  or  low  the  upper  end  might  be.  Of  the 
witnesses  who  addressed  quality  directly,  most  minority  witnesses 
said  that  certain  NSF-supported  projects  were  poor;  all  other  wit- 
nesses said  NSF-supported  projects  are  generally  good. 

The  following  quotes  from  the  record  reflect  witnesses'  direct 
statements  about  the  usefulness  of  peer  review.  Many  of  the  witnesses 
quoted  below  also  suggested  changes  in  the  Foundation's  use  of  peer 
review,  but  no  witness  made  statements  contradictory  to  those  below. 

Raymond  Bowers: 

The  assessment  and  maintenance  of  quality  is  of  fundamental  importance  to 
scholarly  activities  and  institutions.  Peer  review  is  the  principal  procedure  used 
for  quality  control.  (459) » 

I  have  experienced  greater  problems  with  a  few  program  officers,  who  have  very 
strong  opinions  about  how  research  should  be  conducted.  The  results  of  peer 
review  act  as  a  restraining  influence  on  such  officials.  (460) 

William  Carey: 

In  its  general  design  and  structure,  as  well  as  its  purposes,  the  peer  review  pro- 
cedures seem  to  me  to  be  a  sensible  and  practical  way  to  validate  the  scientific 
merit  of  research  proposals  ...  I  believe  that  our  expectations  of  peer  review  as 
to  efficiency,  reliability,  and  quality  control  should  be  tempered  (however)  by 
realism.  (774,  775) 

1  Numbers  in  parentheses  following  quotes  are  page  numbers  of  the  pages  in  the  hearing 
record  where  the  quotes  are  to  be  found. 
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Norman  Hackerman: 


Review  by  peers  brings  to  the  (National  Science)  Board's  oversight  function, 
as  well  as  to  staff  review,  distinct  benefits  in  judging  the  quality  of  the  proposed 
research  and  the  qualifications  of  the  principal  investigator.  It  strengthens  the 
decision  process  by  providing  the  knowledge  of  the  scientists  working  at  the 
forefront  in  the  field  to  complement  the  competency  of  the  staff  of  the  Foundation 
and  the  Board.  (74,  75) 

Philip  Handler: 

Whereas  programmatic  decisions  with  respect  to  the  details  of  programs  of 
technology  development  and  of  most  programs  of  applied  research  must  be  made 
by  mission-conscious,  agency  program  managers,  the  principal  directions  and 
thrusts  of  federally-supported  fundamental  research  are  necessarily  dictated 
largely  by  the  progress  of  science  itself,  as  perceived  by  scientists  and  expressed 
in  the  research  proposals  of  individual  scientists.  (987) 

Virtually  all  who  have  participated  in  or  examined  the  operation  of  the  research 
project  grant  award  system  have  concluded  that  it  is  critically  dependent  upon 
the  use  by  the  agency  staff  of  expert  advisors  to  help  identify  the  scientifically 
most  meritorious  projects  for  support.  No  matter  how  knowledgeable,  sophisti- 
cated or  talented  the  federal  administrator,  even  at  the  time  of  his  initial  appoint- 
ment, it  is  highly  unlikely  that  he  can,  unaided,  adequately  evaluate  and  compare 
the  merits  of  the  diverse  research  proposals  which  will  confront  him.  (990) 

As  we  have  seen,  it  would  be  difficult  to  support  a  contention  that  the  peer 
review  system,  as  practiced  at  the  NSF,  has  not  performed  well  over  the  past  25 
3^ears  .  .  .  Yet,  one  senses  a  growing  concern  that  the  peer  review  system  be 
examined  and,  perhaps  overhauled.   (991) 

Vincent  Haneman : 

Second,  the  peer  review  system  used  to  evaluate  proposals  has  a  number  of 
problems,  but  it  does  provide  a  strong  motivation  and  drive  toward  excellence. 
As  far  as  I  have  been  able  to  determine,  it  is  far  better  than  any  other  method  we 

can  devise.  (614) 

Charles  Kidd : 

Colleges  and  universities  strongly  support  the  peer  review  system  for  a  number 
of  reasons.  (472) 

In  conclusion  and  in  response  to  the  basic  question  as  to  the  worth  of  the  system, 
peer  review  is  a  proven  and  valuable  part  of  the  American  scientific  scene.  It 
should  continue.  (475) 

Gilbert  Ling : 

The  third  part  is  to  emphasize  that  peer  review  is  not  just  a  lot  of  faults  and 
no  merit.  Indeed,  it  is  a  good  system  in  many  ways.  .  .  .  What  is  really  seriously 
missing  is  an  appeal  system.  (877) 

James  Powell: 

The  principal  advantage  of  the  peer  review  sj^stem  is  that  research  proposals  are 
judged  on  their  scientific  merit  by  those  who  have  the  most  information  on  which 
to  base  their  judgement:  practicing  research  scientists.  (428) 

Guyford  Stever: 

I  still  remain  convinced  that  NSF's  peer  review  system  is  a  fundamentally 
sound  method  for  providing  a  broad  base  of  scientific  judgement  in  the  decision 
process.  (901) 

Doris  Wilsdorf : 

I  am  a  strong  advocate  of  the  .  .  .  peer  review  system  ...  I  am  trying  to 
convey  so  strongly  that  if  the  peer  review  system  is  objectively  administered,  it 
can  do  beautifully.  If  not,  then,  obviously,  it  can  do  anything.  (748) 

2.  Innovative  Research 

Many  witnesses  and  Congressmen  attested  to  their  conviction  that 
innovative  research  should  be  supported.  No  one  spoke  against  this 
notion. 
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While  many  witnesses  avowed  that  peer  review  results  in  the  support 
of  high-quality  research,  some  of  which  is  truly  innovative,  there  was 
not  much  confidence  expressed  that  peer  review  consistently  leads  to 
the  support  of  innovative  research  if  it  challenges  the  mainstream  of 
scientific  thought  or  if  it  seems  unlikely  to  succeed.  Arguments  and  the 
weight  of  opinion  to  the  contrary  were  rather  persuasive. 

Since  no  objective  data  were  presented  pertinent  to  this  issue — 
largely,  it  is  believed,  because  such  data  are  rare  and  hard  to  inter- 
pret— the  Subcommittee  has  had  to  rely  on  argument  and  opinion. 

The  argument  that  the  National  Science  Foundation  does  support 
innovative  research  is  that  program  managers  seek  to  fund  excellent 
proposals,  and  "nobody  wants  to  turn  down  the  next  Nobel 
prizewinner." 

Arguments  that  peer  review  does  not  lead  predictabty  to  the  support 
of  innovative  research  if  it  is  challenging  or  risky  are  roughly  as  follows: 

If  a  proposal  challenges  the  mainstream  of  scientific  thought,  the 
expert  peer  reviewer  who  is  the  mainstream  will  tend  to  see  the  proposal 
as  wrong  on  the  face  of  it.  The  expert  reviewer,  in  particular,  is  likely 
to  have  worked  on  experiments  which  either  assumed  the  mainstream 
hypothesis  to  be  correct  or  which  tested  and  corroborated  the  hypoth- 
esis. The  reviewer's  self  esteem  and  reputation  in  the  scientific 
community  may  therefore  depend  upon  the  correctness  of  the  main- 
stream of  thought.  Moreover,  the  reviewer's  laboratory  facilities  and 
Foundation  grant  may  be  proven  useless  if  the  innovative  hypothesis  is 
correct.  Thus,  it  may  be  contrary  to  the  reviewer's  interest  to  find 
merit  in  the  proposal. 

Evidence  in  the  form  of  personal  experience,  purporting  to  show 
that  the  Foundation  failed  to  support  a  correct  but  challenging 
innovative  theory  was  given  by  Drs.  Ling,  Hazlewood,  and  Cope. 
The  scientific  dispute  on  which  their  testimony  was  based  has  not  yet 
been  resolved  in  the  scientific  community. 

Evidence  supporting  the  argument  that  the  scientific  community 
believes  peer  review  often  fails  to  lead  to  the  support  of  challenging 
innovative  research  was  presented  by  two  witnesses  in  the  form  of 
separate  Industrial  Reseaich  Magazine  polls.  While  both  polls  are 
extremely  suspect  because  of  the  miniscule  response  rates  (less  than 
5%),  and  selection  of  potential  respondents  by  criteria  completely 
different  from  those  one  would  use  to  obtain  a  cross-section  of  the 
scientific  community  (the  criterion  was  that  the  respondent  received 
the  free  magazine  Industrial  Research),  no  evidence  to  the  contrary 
was  introduced  or  even  mentioned. 

Some  witnesses  said  that  proposals  combining  high  potential 
scientific  payoff  with  low  chance  of  success  should  be  funded  selectively 
but  would  be  underfunded  because  both  program  managers  and  peer 
reviewers  would  be  unwilling  to  risk  failure.  This  would  occur  whether 
or  not  the  proposals  were  challenging  to  the  mainstream  of  thought, 
witnesses  implied. 

The  issue  of  innovative  research  is  not  self-contained.  It  is  tied 
strongly  to  (1)  appeals  to  Foundation  award  decisions,  (2)  overlap 
of  research  areas  supported  by  Federal  agencies,  and  (3)  Congressional 
review  of  research  grants.  The  arguments  are: 

1 .  An  appeals  process  would  help  insure  that  worthy  innovative 
proposals  are  funded. 
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2.  As  long  as  several  Federal  agencies  are  supporting  research  in  a 
field,  chances  are  that  at  least  one  of  them  will  support  a  worthy 
innovative  proposal.  This  has  happened  in  the  case  of  Dr.  Ling,  who 
was  funded  l>v  NIH  after  the  Foundation  turned  down  his  proposal. 

3.  Congressional  review  of  research  grants  would  likely  stifle 
innovation. 

Further  explication  of  these  arguments  is  given  in  other  parts  of 
this  analysis  of  testimony. 

Need  for  Innovative  Research 

William  Care}^: 

I  believe  it  (peer  review)  should  seek  to  strengthen  American  science  and  our 
ideas  of  opportunity  and  fairness,  by  providing  some  incentives  for  reviewers  to 
look  for  new  and  promising  scientists  just  entering  the  research  profession,  and 
for  new  ideas  which  involve  genuine  risk  taking.   (782) 

Philip  Handler: 

As  has  been  so  frequently  noted,  "In  science,  the  best  is  vastly  more  important 
than  the  next  best."  One  cannot  sum  several  mediocre  projects  in  order  to  gain  the 
equivalent  of  one  superlative  effort.   .   .   . 

The  scientists  in  the  very  good  and  good  categories  greatly  outnumber  those 
whom  we  might  regard  as  Excellent.  Their  research  contributes  the  bricks  and 
mortar  with  which  the  house  of  science  is  built;  the  relative  handful  of  Excellent 
scientists  are,  in  effect,  its  architects.  (989) 

Guy  ford  St  ever: 

A  number  of  witnesses  at  these  hearings  have  expressed  concern  that  the  system 
may  be  biased  against  the  really  innovative  proposal,  one  that  goes  against  the 
tradition  in  its  field  and  which  could  lead  to  a  major  conceptual  breakthrough. 
This  is  a  concern  that  I  share,  as  do  the  NSF  program  officers  and  most  reviewers 
of  proposals.  No  one  would  like  to  think  he  had  turned  down  or  recommended 
turning  down  the  next  Nobel  laureate.  (903) 

Sujjport  oj  Innovative  Research 

Richard  Atkinson: 

I  do  feel  that  the  maverick  in  our  system,  the  person  who  really  has  an  idea 
that  is  counter  to  what  most  of  the  scientific  community  believes,  is  going  to 
receive  very  special  treatment  in  our  system.  If  anything,  I  think  our  sj'stem  leans 
over  a  little  too  far  in  the  direction  of  trying  to  favor  the  maverick  ...  I  think 
Copernicus  or  Galileo  would  have  been  sought  out  by  program  officers  at  the 
National  Science  Foundation  and  would  have  been  funded,  but  they  would  have 
been  watched  very  carefully.  (285) 

Robert  Bauman : 

Is  it  possible,  in  other  words,  that  NSF  policies  actually  discourage  innovative 
and  creative  scientific  efforts?  ...  in  the  correspondence  I  have  received  there 
have  been  a  number  of  instances  of  charges  from  some  of  NSF's  past  recipients, 
at  least  the}'  identified  themselves  as  such,  saying  just  about  what  I  passed  along 
to  you,  that  new  ideas  are  being  stifled  by  NSF.  (202) 

William  Carey: 

I  suspect  that,  under  severe  rationing  conditions,  risk-taking  tends  to  suffer. 
Whether  my  suspicion  is  well-founded,  I  am  unable  to  say,  because  nobody 
seenis  to  analvze  the  output  of  the  peer  review  process  with  this  question  in 
mind.  (776) 

Carlton  Hazelwood: 

The  present  method  of  peer  review  contains  in  my  opinion,  a  natural  bias 
against  revolutionary  ideas  and  findings.  In  my  testimony  I  present  three  major 
criticisms  of  our  system  and  present  a  possible  solution.  One,  grant  proposals 
are  evaluated  in  terms  of  established  concepts.  Two,  peer  reviewers  are  not 
neutral  parties  necessarily  but  have  vested  interest  in  the  outcome  .  .  .  three, 
no  higher  court  of  appeals  exists.  (863) 
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B.    MANAGING   PEER   REVIEW  AT  THE   FOUNDATION 

At  the  outset  it  must  be  observed  that  any  system  used  by  the 
Foundation  for  evaluating  applications  for  scientific  awards  is  apt  to 
displease  many  competent  scientists.  The  problem  was  clearly  enun- 
ciated by 

Philip  Handler: 

(A)  growing  fraction  of  all  worthwhile  research  proposals  is  rejected  annually 
for  lack  of  funds  rather  than  lack  of  scientific  merit.  Under  such  circumstances, 
it  is  inescapable  that  there  will  be  competent  scientists  complaining  that  the  peer 
review  process  did  them  an  injustice  and  alleging  bias  or  other  shortcomings  in 
the  system.  (992) 

Another  facet  of  this  problem  is  that  if  a  project  cannot  be  executed 
for  lack  of  funds,  the  thought  and  time  that  went  into  the  project's 
planning  is  wasted. 

Rustum  Roy: 

I  think  today  that  the  .  .  .  peer  review  system  is  approaching  a  dangerous  level 
of  inefficiency.  We  waste  more  time  using  up  the  best  brains  of  the  country  in 
writing  their  own  proposals.  (754) 

Both  of  the  above  quotes  would  extend  to  any  evaluation  system 
for  project  proposals.  Research  support  outside  the  project  SArste;ii 
might  avoid  project  evaluations  altogether,  but  the  Subcommittee 
believes  project  support  (as  opposed  to  blanket  funding,  or  supporting 
individuals)  is  still  far  the  best  mode  for  distributing  the  bulk  of 
Foundation  funds. 

1.  Management  Policy 

Not  much  testimony  was  explicitly  directed  towards  the  matter  of 
who  should  set  policies  for  peer  review  at  the  National  Science  Founda- 
tion. The  Director  of  the  Foundation,  it  may  be  inferred,  is  a  principal 
actor  in  policy  determination.  National  Science  Board  members 
Norman  Hackerman  and  Donald  Rice  noted  the  responsibilities  of  the 
Board  for  setting  policy  and  explained  the  Board's  present  policy 
interests.  Congressmen  John  Conlan  and  Robert  Bauman  made  it 
clear  they  felt  Congress  should  play  a  strong  role  in  policy  determina- 
tion. All  other  Congressmen  who  spoke  to  the  issue  felt  Congress 
should  be  somewhat  more  distant.  A  further  look  at  testimony  con- 
cerning Congressional  involvement  appears  later  in  this  analysis  of 
testimony. 

William  Carey: 

I  do  not  suggest  a  "manual  of  instructions"  because  I  think  such  things  are 
generally  deadly  and  counterproductive,  but  I  do  believe  that  top  management 
should  make  clear,  in  writing,  the  role  of  peer  review  in  decision-making  and  its 
expectations  as  to  the  uses  and  the  limts  of  peer  review.  This  is  the  very  least  it 
can  do,  and  there  is  every  reason  for  sharing  these  statements  with  the  scientific 
public  so  that  each  side  knows  where  it  stands.  (781) 

Norman  Hackerman: 

The  National  Science  Board  was  charged  by  the  Act  (governing  the  Foundation) 
with  responsibility  for  establishing  the  policies  of  the  Foundation  to  carry  out 
.    .    .    provisions  of  the  Act.  (71) 

(T)he  Board  reviews  from  time  to  time  the  award  decision-making  procedures  of 
the  Foundation,  including  the  solicitation  of  opinions  from  the  scientific  peers  of 
the  investigator  requesting  support,  the  role  of  the  scientific  staff  of  the  Founda- 
tion, and  the  role  of  the  Board  itself.  (76) 
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Philip  Handler: 

On  the  other  hand,  it  would  be  useful  if  the  National  Science  Board  would  pre- 
pare  a   comprehensive   statement   of   the   philosophy,    policies,    and   procedures 
rning  the  NSF  peer  review  system,  a  statement  that  would  be  informative 
and  instructive  to  the  grant  ''applicant,  the  Congress  and  the  interested  public. 
(99S,  999) 

Donald  Rice: 

The  Board  is  continuing  its  examination  and  oversight  of  the  award  decision- 
making process  with  the  intent  of  making  it  the  most  effective,  fair,  and  credible 
system  we  can  devise.  (769) 

James  Symington: 

The  role  of  Congress  and  this  committee  is  and  has  been  to  review  the  program 
and  budgets  of  the  executive  agencies.  Necessarily,  the  emphasis  in  this  process 
is  upon  the  major  objectives  of  managerial  performance  and  program  achieve- 
ment. (878) 

2.  NSF  Staff 

a.  Role 

The  Foundation's  staff  plays  a  key  role  in  the  decision-making 
process  for  individual  awards.  Staff  responsibilities  characteristic 
of  most  Foundation  decision-making  are : 

(1)  Selecting  reviewers  and  soliciting  reviews. 

(2)  Using  the  reviews  as  advice. 

(3)  Making  the  initial  funding  decision  on  a  proposal  and 
reviewing  the  decision. 

There  are  extremes  of  difference  within  the  Foundation  from  this 
set  of  responsibilities  (for  example  the  processes  used  for  graduate 
fellowships,  international  travel  grants,  and  national  research  centers 
are  quite  dissimilar),  but  the  bulk  of  decision-making  finds  the  staff 
in    the    characteristic    role. 

From  the  witnesses'  testimony  it  may  be  concluded  that  the  role 
of  Foundation  staff  members  described  above  has  proved  practicable 
in  that  it  does  result  in  decisions  being  made,  expert  advice  has  been 
used  in  the  process,  and  comparatively  few  complaints  about  the  out- 
come of  the  process  have  been  voiced.  There  were  many  suggestions 
for  changes  in  the  staff  role,  however.  These  are  discussed  later 
in  thi-  report. 

Because  the  Foundation's  staff  has  been  a  key  determinant  of  the 
Foundation's  performance  heretofore,  and  changes  in  staff  responsi- 
bility lead  to  changes  in  staff  morale  in  the  short  run  and  staff  capa- 
bility in  the  long  run,  it  is  evident  that  the  effects  on  the  staff  of  any 
redistribution  of  responsibility  should  be  carefully  considered  and 
monitored. 

b.  Overload 

Several  witnesses  noted  the  size  of  the  workload  on  Foundation 
staff  members  and  said  it  represents  an  overload  in  that  it  would  not 
be  possible  to  give  sufficient  attention  to  each  proposal.  In  the  basic 
research  directorate,  for  example,  a  program  director  will  act  on  50  to 
250  proposals  annually.  Section  heads,  the  immediate  superiors  of 
program  directors,  review  in  the  neighborhood  of  300  recommenda- 
tions to  fund  proposals  each  year. 

The  Foundation  observes  a  continuing  general  increase  in  the 
quality  of  proposals  it  receives.  Whereas  separating  the  wheat  from 
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the  chaff  sufficed  in  former  years,  in  the  face  of  better  proposals  and 
limited  funds,  it  is  now  a  question  of  separating  the  good  from  the 
excellent,  a  more  time-consuming  task.  As  a  result  there  is  frequently 
a  lone:  delay  between  the  submission  of  a  proposal  and  a  determination 
of  whether  it  will  be  funded.  The  number  and  length  of  delays  may 
become  a  serious  problem,  or  perhaps  the  problem  already  exists. 

Xo  witness  suggested  an  increase  in  the  size  of  the  Foundation  staff  to 
counteract  the  overload  problem. 

William  Carey: 

Times  have  changed.  The  traffic  of  proposal  business  now  clogs  the  arteries,  and 
in  trying  to  move  this  traffic  there  are  bound  to  be  heavy  strains  od  the  system. 
Matters  are  further  complicated  by  the  need  to  ration  limited  funds,  and  this 
intensifies  the  problems  of  choices  and  equity.  The  expansion  and  diversification 
of  science,  fed  for  decades  by  Government  stimulation,  has  had  the  effect  of 
multiplying  opportunities  for  good  research  and  the  number  of  scientists  capable 
of  doing  the  work.  Budget  economies  have  denied  agencies  like  NSF  the  man- 
power to  stay  on  top  of  workloads  and  to  monitor  ongoing  research,  and  at  the 
same  time  navigate  the  sea  of  new  proposals  and  renewals.  All  of  these  considera- 
tions cause  me  to  suspect  that  the  peer  review  system  is  under  far  more  pressure 
than  it  was  designed  for,  and  the  only  surprise  is  that  it  has  not  collapsed.  On  the 
whole,  it  has  held  up  extremely  well.  (775) 

c.  Performance 

Testimony  concerning  the  honesty  and  competence  of  the  Founda- 
tion's staff  was  mixed  but  on  the  whole  moderately  supportive. 
Rep.  Conlan  was  extremely  critical  of  the  staff  in  these  respects.  His 
views  were  supported  only  by  certain  of  the  witnesses  he  asked  to 
testify.  Other  witnesses  generally  held  the  Foundation's  staff*  in  high 
regard.  Letters  received  by  the  Subcommittee  tended  to  be  supportive 
of  the  Foundation's  staff. 

It  was  suggested  by  Rep.  Conlan  that  Foundation  program  mana- 
gers know  the  reviewers  they  use  well  enough  for  the  manager  to 
get  any  re-ponse  he  wants  on  a  proposal  by  selecting  the  right  reviewer. 
Several  witnesses  addressed  this  specific  point  in  response  to  Members' 
questions.  The  result  was  that  all  who  expressed  an  opinion  thought 
it  would  be  possible  for  program  managers  to  do  this,  but  so  far  as  any 
witness  knew,  it  was  not  being  done.  Several  witnesses  felt,  however, 
that  the  possibility  of  this  type  of  manipulation  should  be  eliminated 
anyway. 

Mr.  Symington : 

The  question  is  whether  the  program  directors  have  programmed  the  entire 
series  of  decisions  through  the  selection  of  the  peer  reviewers. 

Dr.  Roy: 

My  position  is  that  it  is  certainly  possible  ...  I  don't  have  the  data  at  this 
time  to  say  this  has  been  done  within  NSF.  (762) 

Mr.  Symington: 

(Do)  program  managers  know  how  to  manipulate  the  reviewers  to  get  the 
answers  back  that  they  want,  by  knowing  who  the  reviewers  are  and  what  biases 
they  have. 

Mr.  Carey: 

I  think  it  is  possible  for  program  managers,  indeed,  to  manipulate  the  sys- 
tem ...  I  have  no  personal  evidence  of  this  happening,  but  I  quite  agree  that 
that  it  is  possible.  (77^ 
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Two  specific  allegations  of  lapses  in  staff  performance  were  made  by 
Rep.  Conlan:  the  case  of  the  Individualized  Science  Instructional 
System  (ISIS)  and  the  proposal  of  Ahmad  Shouman. 

d.  ISIS 

The  essence  of  the  allegation  in  the  ISIS  case  is  that  the  Founda- 
tion's staff  willfully  misrepresented  the  peer  reviewers'  comments  on 
the  ISIS  proposal  when  the  proposal  was  presented  to  the  National 
Science  Board  in  order  to  ensure  that  the  Board  would  approve.  The 
implication  was  made  that  the  proposal  would  not  have  been  funded 
if  the  reviewers'  comments  had  not  been  misrepresented. 

The  Subcommittee  is  conducting  an  investigation  of  the  ISIS  case 
through  its  staff  and  with  the  assistance  of  the  General  Accounting 
Office.  The  investigation  is  not  yet  complete.  The  Subcommittee 
plans  to  make  a  statement  about  its  investigation  of  ISIS  at  a  later 
date. 

e.  Shouman 

The  basic  point  made  concerning  the  Shouman  proposal  was  that 
the  Foundation's  staff  officer  handling  the  proposal  acted  autono- 
mously in  a  manner  not  in  accordance  with  then-existing  Foundation 
policy.  The  Foundation  admits  that  the  staff  officer  did  send 
verbatim  reviews  to  Dr.  Shouman,  which  at  the  time  was  contrary 
to  policy.  The  staff  officer  involved  was  transferred  out  of  the  Founda- 
tion some  time  ago  in  a  transfer  having  no  connection  with  the 
Shouman  case. 

3.  Reviewers 

a.  Role 

The  role  of  peer  reviewers  used  by  the  National  Science  Foundation 
is  straightforward  and  has  proved  practicable  during  the  last  25  years. 

An  individual  peer  reviewer  normally  receives  in  the  mail  a  single 
proposal  to  review.  The  reviewer  then  studies  the  proposal  and  other 
material  if  necessary,  writes  a  review  of  one  or  a  few  pages,  and  mails 
the  review  back  to  the  Foundation  with  the  (possibly  marked) 
proposal. 

The  review  of  a  proposal  will  normally  take  into  account  the 
technical  adequacy  of  the  proposal,  the  importance  to  science  (or  to 
the  solution  of  a  real-world  problem)  of  the  proposed  work,  and  the 
likelihood  of  success  by  the  proposer  (generally  called  the  principal 
investigator  (PI)).  Past  performance  of  the  proposer  is  taken  into 
account,  particularly  in  judging  the  likelihood  of  success. 

Peer  panel  reviewers  normally  each  receive  in  the  mail  a  number  of 
proposals  in  one  batch  to  review.  Frequently  a  particular  reviewer 
will  be  asked  to  read  all  proposals  in  a  batch,  but  will  only  review 
selected  ones  in  depth.  Following  this  private  review,  the  panel  will 
meet  and  each  proposal  will  be  discussed.  A  rating  of  each  proposal 
by  the  panel  as  a  whole  will  normally  be  generated — either  by  panel 
agreement  or  by  some  balloting  procedure.  Depending  upon  the  format 
of  the  panel  meeting,  the  panel's  rating  may  be  morally  binding  on  the 
program  manager  to  a  greater  or  lesser  extent. 

b.  Merit 

The  outstanding  merit  of  employing  peer  review  for  the  evaluation 
of  applications  for  scientific  awards  is  that  the  highest  available 
expertise  is  applied  to  the  evaluation.  The  scientific  community  has 
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accepted  peer  reviewing  as  a  professional  obligation.  The  Foundation 
reports  that  92  percent  of  the  reviews  it  requests  are  performed.  Peer 
reviewers  are  normally  reimbursed  by  the  Foundation  for  expenses 
incurred  in  reviewing  (usually  nothing  for  individual  reviewers,  travel 
and  living  expenses  for  panel  reviewers  who  must  get  together),  but 
normally  receive  no  payment  for  their  reviewing  services.  The  value 
of  an  average  review,  were  it  paid  for  at  a  rate  reviewers  might  charge 
if  they  were  acting  as  consultants,  would  likely  lie  between  $20  and 
$100.  The  value  of  the  roughly  100,000  reviews  contributed  free  to 
the  Foundation  by  the  scientific  community  each  year  is  thus  in  the 
range  of  $2,000,000  to  $10,000,000.  Reviewers  do  derive  non-monetary 
benefits  from  reviewing,  but  these  are  impossible  to  quantify.  One 
benefit  is  that  the  reviewer  discovers  the  research  ideas  of  someone 
else  in  the  field.  Over  an  entire  scientific  subcommunity  the  reviewing 
system  thus  acts  as  a  DEW  line  for  scientific  discoveries.  A  second 
benefit  to  the  reviewer  is  that  the  Foundation  program  manager 
learns  something  about  the  reviewer's  capabilities  and  performance. 

c.  Bias 

The  principal  drawback  to  the  use  of  peer  review  for  the  evaluation 
of  applications  for  scientific  awards  lies,  as  several  witnesses  said,  in 
the  potential  for  biased  evaluations  by  the  reviewers.  The  most 
frequently  cited  form  of  potential  bias  arises  from  the  existence  of 
"old  boys  clubs,"  groups  of  people  in  a  discipline  who  know  each  other. 
It  is  alleged  that  members  of  such  groups  will  praise  each  others' 
proposals  unduly,  and  downgrade  outsiders'  proposals  unduly.  This 
behavior  might  be  either  intentionally  self-serving  or  the  result  of  a 
view  of  the  discipline  common  to  the  group's  members. 

Other  forms  of  reviewer  bias  mentioned  by  witnesses  included  bias 
against  someone  a  reviewer  knew  and  disliked  or  disagreed  with,  or 
favoritism  towards  persoDal  acquaintances  to  the  disadvantage  of 
strangers  simply  by  virtue  of  being  able  to  make  a  better-informed 
judgment. 

Neither  the  problem  of  bias  nor  the  following  question  of  overconcen- 
tration  of  reviewing  is  easily  addressed.  Despite  wide  agreement  that 
"old  boys  clubs"  could  occur  and  probably  had,  not  a  single  example  of 
an  "old  boys  club"  was  mentioned. 

The  reasons  for  difficulty  in  determining  bias  are: 

1.  Most  importantly,  talent  is  concentrated.  In  most  fields  of  science 
there  are  a  handful  of  acknowledged  centers  of  excellence.  Talented 
people  frequently  move  to  such  centers,  and  the  centers  actively  re- 
cruit talent  from  wherever  they  can  find  it.  Not  surprisingly,  reviewers 
at  these  centers  tend  to  rate  proposals  from  other  centers  highly. 

2.  In  any  scientific  specialty  practitioners  of  the  specialty  are  ex- 
tremely likely  to  know  one  another  and  have  some  easily  describable 
relationship  such  as:  "They  both  went  to  Stanvard"  or  "their  advisors 
were  roommates  at  Harford." 

3.  Reviewing  scientific  proposals  is  an  extremely  subjective  matter. 
It  is  not  possible  to  take  a  proposal,  stick  it  into  a  machine,  and  get 
a  quality  rating.  If  it  were  possible,  peer  reviewing  would  not  be 
necessary. 

d.  Distribution 

A  second  question  about  the  Foundation's  use  of  reviewers,  implicit 
in  the  testimony  of  some  witnesses,  is  "Does  the  Foundation  use  all  the 
competent  potential  reviewers."  It  was  agreed  by  all  who  addressed  the 
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question  that  reviewing  should  not  be  concentrated  in  the  hands  of  a 
small  portion  of  the  competent  community.  The  obverse  of  this  point 
i<  that  neither  should  the  reviewing  be  spread  so  thinly  that  no  scientist 
can  become  a  proficient  reviewer.  Reviewing,  like  most  other  human 
-kills,  may  be  improved  through  practice. 

The  concentration  of  talent  mentioned  above  is  also  the  principal 
difficulty  in  judging  the  appropriateness  of  the  distribution  of 
reviewing. 

Data  presented  by  the  Foundation  concerning  the  utilization  of  re- 
viewers was  highly  aggregated  and,  therefore,  provided  only  modest 
illumination  of  the  issues.  The  Subcommittee  believes  the  data  do  show 
that  reviewing  for  the  Foundation  is  not  intensely  concentrated  but  do 
not  reveal  whether  the  distribution  of  reviewing  is  sufficiently  broad. 
Xo  one  has  suggested  it  might  be  too  broad  already.  The  Foundation's 
300  or  so  program  managers  used  about  30,000  different  reviewers  in 
Fiscal  Year  1974  who  performed  about  120,000  reviews  altogether. 
The  number  of  science  and  engineering  doctorate  holders  in  the  United 
States  is  about  240,000. 

Roughly  speaking,  then,  each  Foundation  program  manager  uses  an 
average  of  100  reviewers  each  year,  each  of  whom  performs  4  reviews, 
but  only  one  in  eight  doctorate  holders  performs  reviews  for  the 
Foundation  each  year.  These  derived  figures  should  not  be  used  or  ex- 
pected to  approximate  the  analogous  figures  for  any  given  Foundation 
program;  the  aggregated  numbers  blur  many  unique  situations. 

No  data  or  other  evidence  was  presented  to  indicate  how  much 
change  there  is  from  year  to  year  in  the  set  of  reviewers  used  by  the 
Foundation.  The  Subcommittee  does  not  doubt  that  the  membership 
of  standing  peer  panels  used  by  the  Foundation  rotates,  but  is  unin- 
formed about  rotation  of  individual  peer  reviewers  and  ad  hoc 
panelists. 

DISTRIBUTION   OF  REVIEWING 

Raymond  Bowers: 

The  recent  decision  of  the  National  Science  Board  to  make  available  lists  of 
reviewers  used  in  the  various  scientific  areas  is  constructive.  It  would  be  desirable 
to  require  the  agencies  to  maintain  a  steady  turnover  of  reviewers.  (461) 

Charles  Kidd: 

Membership  on  peer  review  groups  or  panels  should  rotate.  No  one  questions  the 
necessity  of  rotation  of  membership  when  formal  "outside"  review  groups  are 
u^ed  and  the  reviewers  are  appointed  for  fixed  terms.  What  is  required  is  a  reason- 
able balance  between  rotation  at  a  relatively  rapid  rate  in  order  to  expand  the 
number  who  serve  as  reviewers  and  rotation  at  a  rate  slow  enough  to  allow  re- 
viewers to  become  familiar  with  the  review  system.  (474) 

James  Powell: 

I  would  like  to  close  with  four  specific  recommendations  which  we  believe  would 
improve  the  peer  review  system.  For  research  proposals:  One,  review  panelists 
should  include  highly  qualified  scientists  from  undergraduate  institutions  as  well 
as  ones  from  graduate  institutions.  (445) 

Several  different  modes  of  reviewing  were  proposed  by  witnesses, 
and  discussed  in  some  detail.  Each  of  these  modes  is  connected  to 
several  issues.  The  suggestions  deserving  serious  consideration  were: 
blind  reviewing,  the  creation  of  reviewer  "stables,"  possibly  combined 
with  the  random  selection  of  reviewers,  and  increased  emphasis  on 
panel  review. 
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e.  Blind  Reviewing 

In  "blind  reviewing"  the  name  of  the  proposer  is  deleted  from  the 
proposal  before  it  is  provided  to  reviewers.  The  principle  on  which  this 
idea  is  based  is  that  proposals  should  be  evaluated  on  the  basis  of  their 
w  ritten  content  only;  it  being  presupposed  that  the  likelihood  of  suc- 
cess of  the  proposed  project  can  be  judged  without  knowing  the 
identity  of  the  proposer. 

Blind  reviewing  is  used  with  some  success  by  a  few  scientific  journals, 
but  as  mentioned  earlier,  a  journal  article  is  a  finished  piece  of  work, 
whereas  proposed  research  is  not.  A  few  witnesses  spoke  in  favor  of 
blind  or  half-blind  reviewing,  but  most  witnesses  were  opposed.  The 
Subcommittee's  analysis  is  that  blind  reviewing  of  proposals  for 
scientific  awards  is  a  procedure  with  little  potential  merit.  The  one 
known  "pro"  and  the  many  "cons"  are: 

PRO 

If  the  blindness  of  a  proposal  can  be  maintained,  the  reviewer  cannot 
play  favorites  or  otherwise  be  biased  by  his  knowledge  or  ignorance  of 
the  proposer. 

CON 

1.  Most  importantly,  the  identity  of  the  proposer  is  a  significant 
factor  in  determining  the  likelihood  of  success  of  a  project.  The  elimi- 
nation of  knowledge  of  this  factor  from  the  ken  of  the  reviewer  is 
likely  to  lead  to  a  decrease  in  the  average  quality  of  research  per- 
formed with  the  Foundation's  assistance. 

2.  Concealing  the  identity  of  the  proposer  from  a  reviewer  would  be 
difficult  in  many  cases.  Proposals  frequently  contain  references  to  the 
proposer's  past  achievements  including  citations  of  papers  written. 
Such  proposals  would  have  to  be  rewritten  or  edited.  A  description  of 
available  experimental  facilities  or  other  pertinent  information  may 
provide  the  tipoff. 

3.  Reviewers  may  refuse  to  participate  if  reviewing  is  blind.  Prob- 
lems of  this  sort  have  arisen  in  blind  reviewing  of  journal  articles. 

4.  The  elimination  of  the  name  of  the  proposer  would  force  the 
reviewer  to  rely  more  heavily  on  the  quality  of  the  proposal  itself,  a 
measure  largely  of  linguistic  rather  than  scientific  ability.  Even 
those  who  spoke  for  blind  reviewing  preferred  to  have  linguistic 
facility  downplayed. 

Raymond  Bowers: 

(I)n  the  appraisal  one  forms  some  sense  of  whether  that  person  has  a  record  that 
suggests  he  can  do  the  things  he  is  proposing  or  the  institution  has  the  structure  to 
make  it  plausible.  You  are  withholding  a  very  important  piece  of  information 
about  the  plausibility  of  getting  something  done  when  you  don't  let  me  know  who 
and  where.  (468) 

Daniel  Drucker: 

There  is  a  further  difficulty  for  the  reviewer  of  a  proposal.  The  merits  of  a  paper 
are  contained  in  it.  They  can  be  judged  without  regard  as  to  who  wrote  it.  The 
additional  essential  question  to  be  answered  honestly  by  the  proposal  reader  is 
whether  the  investigator  is  competent  to  do  the  actuarresearch  proposed.  (622) 

Heinz  Wilsdorf : 

In  fact,  it  is  most  likely  that  many  a  proposer  in  rich  universities  gets  very 
effective  help  in  the  preparation  of  his  proposals,  .  .  .   (I)t  should  be  made 
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impossible  for  proposals  from  different  institutions  to  already  have  a  built-in 
advantage  or  handicap  depending  on  the  technical  production  thereof,  including 
photographs,  binders,  quality  of  typing,  and  so  forth  .   .  . 

Next,  in  order  to  insure  objectivity  of  the  review,  the  proposer  should  be 
anonymous.  (738) 

/.  Reviewer  Stables  and  Randomization 

Lists  of  approved  potential  reviewers  would  be  generated  by  the 
Foundation  (without  any  particular  proposal  in  mind)  in  a  way  to 
insure  that  all  people  on  the  lists  are  competent.  It  was  suggested  by 
one  witness  that  people  on  the  list  in  a  given  field  might  be  approved 
by  a  professional  society  in  the  field.  Witnesses  appeared  to  believe 
that  the  lists  should  be  published  or  at  least  available  to  the  public, 
though  this  point  was  not  specifically  addressed. 

The  lists  might  be  utilized  in  various  ways.  Most  witnesses  who 
discussed  lists  did  so  in  the  context  of  random  selection.  Alternatively, 
Foundation  program  managers  might  simply  be  required  to  choose 
from  the  list. 

In  random  selection,  for  a  given  proposal,  a  large  list  of  reviewers  is 
selected  and  reviewers  are  chosen  at  random  from  the  list  to  review 
the  particular  proposal. 

Witness  opinion  about  random  selection  was  mixed  but  more 
negative  than  positive  overall. 

Some  pros  and  cons  are: 

PRO 

1.  Random  selection  would  largely  eliminate  the  possibility  of  the 
program  manager  purposely  biasing  the  review  through  selecting 
reviewers  whose  opinions  he  can  predict. 

2.  The  existence  of  a  public  list  of  potential  reviewers  might  assure 
the  applicant  and  the  public  that  objective  decision-making  is  taking 
place. 

CON 

1.  Most  importantly,  the  potential  for  a  review  by  the  most  knowl- 
edgeable people  in  a  carefully  balanced  reviewer  group  would  be 
seriously  diminished.  It  is  unlikely  that  any  randomizing  procedure, 
however  carefully  designed,  could  result  in  an  overall  review  of  as 
high  quality  as  could  be  obtained  by  purposive  selection  of  reviewers. 

2.  Administering  and  auditing  random  selection  would  impose  an 
added  workload  on  the  Foundation,  the  proportions  of  which  are 
likely  to  be  significant. 

3.  The  lists  of  potential  reviewers  would  be  a  source  of  contention. 
There  would  be  disagreement  on  the  criteria  for  putting  people  on 
the  lists  or  removing  them. 

William  Carey: 

I  think  part  of  the  responsibility  of  the  program  manager  is  to  use  his  head, 
and  in  using  his  head,  he  has  to  look  at  the  identities  of  the  peer  reviewers.  (778) 

I  think  if  you  blindfold  them  and  compel  them  to  play  a  randomization  game, 
then  you  are  going  to  miss  an  awful  lot.  You  are  going  to  do  something  to  diminish 
the  responsibility  of  the  government  employee.  And  I  think  that  diminished 
responsibility  in  government  employees  is  not  the  way  we  should  be  running  the 
government.'  (788) 

Donald  Rice: 

The  idea  of  a  randomized  kind  of  selection  procedure  is  one  alternative  which 
does  have  some  merits  in  building  in  a  measure  of  objectivity  in  that  selection 
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process.    I    think   it   is    only   one   of   a   number   of   things    which   ought   to    be 
considered  ....  (769) 

I  would  also  like,  at  this  time,  not  to  be  put  down  as  a  fan  of  random  selection. 
I  think  that  subject  needs  further  examination.  (788) 

Heinz  Wilsdorf : 

I  propose  that  the  reviewers  be  chosen  by  lot  or  by  some  mechanism,  and  that 
this  would  be  a  random  mechanism.  (747) 

g.  Peer  Panels 

The  most  cogently  supported  suggested  change  in  the  use  of  peers  to 
assist  in  evaluating  grant  applications  at  the  National  Science  Founda- 
tion is  to  increase  reliance  on  peer  panels. 

There  is  a  successful  paradigm  for  the  use  of  peer  panels,  their  use 
at  XIH,  described  earlier  in  this  report.  The  strongest  arguments  for 
peer  panel  review  at  the  Foundation  were  set  forth  by  Philip  Handler, 
former  chairman  of  the  National  Science  Board,  who  proposed 
standing  panels  with  rotating  membership,  viz.: 

(1)  Such  an  arrangement  avoids  the  possibility  of  personal  arbitrariness  on  the 
part  of  program  managers.  .  .  .  the  appearance  ...  of  such  behavior  is  avoided 
by  a  formal  meeting  which  leaves  behind  its  minute^,  with  general  rank  ordering 
of  the  proposals  considered,  and  with  a  summary  of  the  group's  opinion  of  each 
application. 

(2)  It  minimizes  the  possibility  of  undue  influence  or  bias  by  individual  re- 
viewers in  giving  preference  to  grant  applicants  from  particular  research- 
performing  institutions  or  from  some  school  of  scientific  thought  since  such 
individual  influences  are  suppressed  or  neutralized  in  the  setting  of  a  panel  meeting. 

(3)  In  the  ad  hoc  mail  review  system,  only  the  program  manager  can  be  influenced 
by  critical  information  known  to  and  cited  by  only  one  of  the  reviewers,  whether 
that  information  influence  the  outcome  favorably  or  unfavorably  for  the  applicant. 
The  staff  decision  will  then  seem  to  be  counter  to  the  recommendations  of  most 
of  the  reviewers.  But  a  panel  or  study  section  meeting  is,  in  effect,  a  superlative 
scientific  seminar.  I  have  witnessed  many  incidents  in  which  a  first-rate  scientist 
has  revised  his  previously  held  judgment  of  the  merits  of  a  particular  proposal 
because  of  information  described  by  one  of  the  other  panel  members.  In  addition, 
such  panel  discussion  frequently  engenders  extremely  helpful  suggestions  which 
can  be  passed  along  to  the  applicant,  thereby  improving  the  quality  of  the  work 
when  it  is  undertaken. 

(4)  By  considering  perhaps  40  to  100  applications  at  a  single,  several  day 
meeting,  the  panel  can  be  of  considerable  additional  assistance  to  the  agency. 
From  their  discussions,  there  can  emerge  a  coherent  strategic  approach  to  research 
in  a  given  area  of  scientific  inquiry,  which,  in  effect,  becomes  agency  policy  in 
that  regard.  By  consecutively  debating  the  merits  of  the  proposals  before  them, 
the  panel  gets  a  better  sense  of  their  relative  intrinsic  merit  and  thus  enables 
their  ordering  so  as  to  mount  the  most  effective  research  effort.  (994) 

Charles  Kidd  and  John  Sherman,  both  of  whom  had  long  experience 
at  NIH,  favored  panel  review. 
Charles  Kidd : 

I  feel  more  confident  and  relaxed  with  a  system  which  relies  heavily  on  panels 
of  reviewers  ...  I  believe  frankly  that  selection  of  a  group  of  advisors  who  rotate 
periodically  and  who  make  a  collective  judgement  on  applications,  in  general 
engenders  a  sense  of  confidence  among  those  being  judged.  The  system  is  one 
where  the  selection  of  reviewers  is  completely  open.  Under  this  system  you  can 
have  disclosure  of  who  made  the  decision  without  any  difficulty.  (477,  478) 

James  Powell  also  ^poke  strongly  in  favor  of  panels,  although  he 
did  not  specify  standing  panels: 

The  .  .  .  review  process  should  include  both  a  mail  review  and  a  meeting  of  the 
review  panelists  in  Washington.  (445) 

The  Foundation  already  makes  fairly  extensive  use  of  panels. 
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Richard  Atkinson: 

(O)f  the  applications  processed  in  1974,  44  percent  had  ad  hoc  reviews  only,  28 
percent,  had  panel  reviews  only,  and  28  percent  had  both  ad  hoc  and  panel  reviews. 
So  .")<;  percent  of  our  proposals  will  have  some  form  of  panel  reviews. 

( Jharles  Mosher: 

(W')hat  determines  which  applicants  have  only  ad  hoc  review  and  which  have 
only  panel  review,  and  which  have  both? 

Richard  Atkinson : 

In  a  certain  sense  the  program  officer  makes  that  decision,  but  by  and  large  it 
i-  historical.  Certain  fields  of  science  have  tended  to  favor  panel  reviews.  Other 
fields  of  science  have  tended  to  favor  ad  hoc  reviews.  (264,  265) 

At  least  one  witness  believed  it  inadvisable  to  increase  reliance  on 
panel  review: 

William  Carey: 

(P)anels  are  in  effect  co-opted  as  adjunct  bodies  and  incorporated  into  the  decision- 
making system,  rightly  or  wrongly.  If  they  sit  as  adjudicators  of  an  array  of 
competing  proposals,  and  rank  order  them  by  a  balloting  procedure,  and  if  their 
rankings  acquire  so  much  weight  that  the  agency  officials  have  in  effect  delegated 
responsibility  to  the  panels,  then  the  equations  have  all  been  changed.  I  happen 
to  believe  that  too  much  accountability  has  been  handed  off  by  such  arrange- 
ments. (780,  781) 

A  second  problem  with  panel  review  (especially  at  the  Foundation 
in  contrast  with  NIH)  is  that  in  many  specialties  it  may  be  impossible 
to  assemble  a  panel  knowledgeable  enough  to  expertly  evaluate  the 
majority  of  proposals  in  the  specialty.  Yet  the  number  of  proposals 
in  the  specialty  may  be  too  small  to  warrant  the  establishment  of 
more  than  one  panel. 

The  total  number  of  reviewers  used  by  the  Foundation  would  likely 
be  decreased  if  peer  panel  review  is  increased.  This  might  have  good 
effects  through  allowing  more  careful  attention  to  the  distribution  of 
reviewers  or  bad  effects  through  the  reduction  itself. 

It  is  clear  that  increased  reliance  on  panel  review  at  the  Foundation 
deserves,  at  the  very  least,  further  consideration  by  the  policy-making 
apparatus  of  the  Foundation. 

4-  Division  of  Responsibility 

a.  General 

Present  decision-making  procedures  at  the  Foundation  find  an 
individual  staff  member  in  a  position  of  central  responsibility  in  many 
programs  at  two  key  points:  the  selection  of  reviewers  and  the  initial 
decision.  Many  witnesses  suggested  changes  that  would  tend  to  reduce 
these  responsibilities.  No  witness  proposed  increasing  the  respon- 
sibilities of  individual  staff  members. 

At  the  point  of  selection  of  reviewers,  a  shift  strongly  away  from 
the  staff  is  evident  if  stables  of  reviewers  are  instituted.  In  the  case 
of  peer  panels  the  staff  would  be  expected  to  retain  discretion  for 
selection  of  panel  members,  although  the  selection  would  be  much  more 
open  to  public  inspection  than  is  now  the  case.  Another  possible 
change  little  mentioned  at  the  hearings,  would  be  to  require  approval  of 
slates  of  reviewers  at  higher  staff  levels  before  proposals  are  sent  out  for 
review;  this  practice  should  be  feasible  for  panel  review,  but  may  add 
a  heavy  burden  if  individual  peer  review  is  used.  The  Foundation 
already  requires  higher  staff  approval  of  reviewers  in  some  programs. 
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In  the  case  of  panel  review,  the  actual  initial  decision  lies  more  with 
the  peers  than  if  individual  peer  review  is  used.  Hence,  increased 
reliance  on  peer  panel  review  would  shift  responsibility  away  from 
the  Foundation's  staff.  More  substantive  review  by  higher  levels  in 
the  Foundation's  staff  would  lessen  the  concentration  of  responsibility 
in  the  individual  staff  member  making  the  initial  decision.  The  Founda- 
tion is  currently  creating  grant  review  boards,  groups  of  staff  members. 
These  boards  will  have  the  effect  just  mentioned. 

Several  witnesses  emphasized  that  the  Foundation  staff  must  retain 
the  right  and  ability  to  go  against  the  recommendations  of  outside 
advisors,  conceding  that  such  action  might  be  rare.  The  crucial  point 
is  that  Federal  employees  must  retain  effective  control  of  the  process 
and  be  able  to  change  it  when  necessary.  No  witness  contended  that 
the  Foundation  had  lost  control  of  the  process. 

A  certain  "conservation  of  staff  responsibility"  principle  may  be 
seen  in  witnesses'  testimony  concerning  panels.  Those  who  favored 
panels  (a  mechanism  limiting  staff  responsibility)  generally  also 
favored  having  the  staff  choose  the  members  of  the  panels.  Charles 
Kidd  and  Philip  Handler  are  the  outstanding  examples: 

Philip  Handler: 

Thus,  although  the  peer  reviewers  should  be  heavily  relied  on  for  judgments  of 
scientific  merit,  agency  ability  to  attract  outstanding  program  managers  will  be 
diminished  if  the  hitter's  responsibility  for  independent  scientific  judgment  is 
largely  removed.  Achieving  a  desirable  balance  between  reliance  on  outside  advice 
and  the  exercise  of  administrative  discretion  should  be  an  important  objective  of 
the  peer  review  management  scheme.  (996) 

b.  Missions 
Several  witnesses  agreed  that  in  the  support  of  mission-oriented 
research  it  is  best  to  keep  basic  responsibilities  in  the  staff  of  an 
agency.  This  notion  was  not  challenged.  The  reasoning  is  that  in  mis- 
sion-oriented research,  since  there  is  a  specific  goal,  it  may  be  possible 
to  tell  what  information  is  needed  to  reach  the  goal,  and  the  responsible 
staff  member  should  be  well-acquainted  with  which  of  the  information 
is  known  and  which  unknown.  Thus  the  staff  member  can  better  judge 
the  key  issue  of  the  importance  of  a  proposed  bit  of  research  in  ful- 
filling the  mission's  objective  than  can  extramural  experts.  This  was 
brought  out  in  the  Foundation's  testimony  concerning  its  evaluation 
methods  in  the  Research  Applications  Directorate,  by  William 
Raney  concerning  the  Office  of  Naval  Research,  and  by  others  to  a 
lesser  extent. 

5.  Policing  the  System 

To  insure  that  the  Foundation's  decision-making  processes  are 
working  properly,  it  is  necessary  that  information  be  available  about 
the  processes  and  that  someone  analyze  the  information.  What  the 
information  should  be,  just  what  analysis  should  be  undertaken,  and 
wTho  should  do  it  were  the  subjects  of  much  testimony. 

Two  loci  of  information  received  particular  attention:  the  "grant 
jackets"  and  a  management  information  system.  Each  proposal 
(other  than  travel  grant  or  fellowship  applications)  normally  has  its 
own  "grant  jacket,"  a  file  containing  at  least  the  proposal,  peer  reviews 
of  the  proposal,  and  a  record  of  decisions  made  on  the  proposal.  Wit- 
nesses agreed  there  should  be  a  document  (or  documents)  in  the  grant 
jacket  outlining  the  rationale  for  whatever  decisions  had  been  made. 
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The  second  locus  of  information  is  a  Foundation  management  informa- 
tion system  which  would  retain,  among  other  things,  the  names  of 
reviewers  and  proposers,  recorded  in  association  with  further  pertinent 
information  (e.g.,  institution,  proposals  reviewed,  proposals  submitted, 
etc.).  The  need  for  a  management  information  system  was  particularly 
evident  in  questions  of  the  distribution  of  reviewing.  The  Foundation 
has  begun  the  establishment  of  a  management  information  system. 

Witnesses'  approaches  to  the  question  of  what  analysis  should  be 
undertaken  (and  to  questions  of  degree  of  openness,  as  is  discussed 
further  on)  revealed  a  fundamental  difference  in  philosophy  among 
witnesses.  One  group,  the  spokesman  for  which  was  Rep.  John 
Conlan,  believed  it  best  to  design  decision-making  systems  defensively, 
i.e.  on  the  presumption  that  the  proportion  of  dishonest  or  unscru- 
pulous people  among  Foundation  program  managers  and  reviewers  is 
high  enough  to  cause  severe  problems  if  those  people  have  a  significant 
opportunity  to  turn  the  sj^stem  to  their  advantage.  The  second, 
larger  group,  the  most  outspoken  member  of  which  was  Rep.  Charles 
Mosher,  believed  that  the  system  should  be  designed  on  the  presump- 
tion that  program  managers  and  reviewers  are,  on  the  whole,  honest 
and  ethical,  but  that  vigilance  should  be  maintained  over  the  system 
in  such  a  way  as  to  insure  that  unscrupulous  acts  are  rare.  The  Sub- 
committee subscribes  to  this  second  viewpoint. 

John  Conlan: 

I  know  from  studying  material  provided  to  me  by  NSF  that  this  is  an  "Old 
Boy's  System,"  where  program  managers  rely  on  trusted  friends  in  the  academic 
community  to  review  their  proposals.  These  friends  recommend  their  friends  as 
reviewers.   .  .  . 

Without  any  effective  management  control  procedures  to  insure  accountability 
in  this  kind  of  system,  it  is  almost  inevitable  that  some  program  managers  may  al- 
most unconsciously  become  advocates  for  certain  scientists  and  their 
projects.  .  .  . 

It  is  an  incestuous  "buddy  system"  that  frequently  stifles  new  ideas  and  scien- 
tific breakthroughs,  while  carving  up  the  multimillion  dollar  Federal  research  and 
education  pie  in  a  monopoly  game  of  grantsmanship.  (5) 

Charles  Mosher: 

It  is  my  personal  belief  that  whatever  mechanism  is  used  to  accomplish  the  job 
that  the  peer  review  system  attempts  to  accomplish,  in  the  long  run  it  all  depends 
on  the  integrity  of  the  people  involved.  (169) 

William  Carey: 

I  think  you  have  to  trust  people.  If  you  hire  good  people  and  you  want  to  make 
them  understand  what  the  public  ethics  are  in  the  management  of  money,  then  I 
think  you  give  them  a  chance  to  do  it  right.  (788) 

Tom  Harkin: 

It  goes  to  the  old  principle  of  trust.  You  have  to  trust  people  ...  it  seems  that 
because  we  don't  trust  people  that  we  have  to  get  more  and  more  police.  I  don'c 
mean  uniformed  police,  more  people  to  check  the  checker  and  somebody  else  to 
check  the  checkers  who  are  checking  the  checkers  because  you  can't  trust  them.  It 
seems  to  me  we  could  have  a  lot  less  government  in  the  country  today  if  people 
trusted  people  a  little  more.  (241,  242) 

The  analysis  to  be  undertaken  in  accordance  with  general  reliance 
on  trust  accompanied  b}r  vigilance  was  suggested  by  several  witnesses. 
It  falls  in  two  parts:  continued  oversight  of  the  broad  issues  (distribu- 
tion of  funds  and  reviewers,  success  ratios,  etc.)  using  information 
largely  contained  in  the  Foundation's  management  information  sys- 
tem, and  random  audits  of  the  decision  processes  for  individual  grants 
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using  information  largely  contained  in  the  grant  jackets.  These  audits 
would  investigate  such  matters  as  ties  between  program  managers  and 
grantees  and  between  reviewers  and  grantees.  Witnesses  who  discussed 
decision  process  audits  believed  these  should  be  internal  to  the  Founda- 
tion and  performed  by  a  group  not  part  of  the  decision-making  ap- 
paratus for  individual  grants  (and  separate  from  the  auditing  functions 
of  the  General  Accounting  Office,  which  should  be  continued).  Over- 
sight functions  of  Congressional  committees  are  discussed  in  the  next 
section. 

William  Carey: 

The  people  at  the  top  of  the  agencies  should  not  assume  that  all  is  well  with 
peer  review  until  they  find  out  differently.  I  do  not  favor  imposing  formidable 
administrative  controls  because  I  believe  this  would  have  a  smothering  effect.  A 
selective,  random  audit  of  staff  practices  regarding  peer  review  is  what  I  have  in 
mind,  and  I  think  it  would  convey  the  proper  message.  (781) 

Vincent  Haneman: 

I  would  like  to  ask  if  there  is  any  way  to  measure  invidious  relationships  be- 
tween groups  of  reviewers  and  groups  of  awardees? 

Now  my  answer  to  that  question  is  yes.  This  is  done  by  an  internal  audit  system 
that  should  be  developed  and  in  operation  by  the  NSF.  I  see  no  reason  at  this 
time  why  this  could  not  be  accomplished  almost  immediately.  (620) 

Donald  Marlowe: 

I  believe  that  it  is  possible  to  have  internal  audit  systems  that  continuously 
monitor  the  flow  of  proposals,  awards  and  provides  solid  data  to  answer  the  ques- 
tions that  have  plagued  the  system  for  years  .  .  .  when  an  audit  system  is 
designed,  it  should  be  kept  separate  from  the  operating  line  organization  ...  I 
don't  want  to  see  all  the  intellectual  effort  of  the  directors  of  these  programs  go 
into  the  monitoring  of  the  system  rather  than  directing  the  system  ...  I  suggest 
that  such  an  auditing  s}rstem  could  probably  be  done  on  a  sampling  basis.  .  .  . 
(618) 

6.  Congressional  Oversight 

The  Subcommittee  had  ample  opportunity  during  the  hearings  to 
explore  whether  Congressional  review  of  individual  National  Science 
Foundation  grants  should  be  required  in  addition  to  Foundation 
approval  before  the  Foundation's  action  becomes  final.  Robert  Bau- 
man,  the  best-known  proponent  of  this  idea,  appeared  as  a  witness. 
Many  other  witnesses  and  Subcommittee  members  addressed  the 
issue  as  well.  Opinion  was  overwhelmingly  against  Congressional 
review. 

The  argument  for  Congressional  review  is  that  the  Foundation  has 
apparently  been  making  a  large  number  of  grants  for  research  of 
dubious  value  and  will  continue  to  do  so  unless  Congress  checks  each 
proposed  grant  to  ascertain  for  itself  the  merit  of  the  proposed  research. 

Arguments  against  Congressional  review  are: 

1.  The  review  of  individual  grants  is  not  the  proper  way  for  Con- 
gress to  exercise  its  oversight  of  an  agency.  Congress  should  not  try 
to  be  part  of  the  day-to-day  operation  of  executive  agencies,  but 
should  review  agency  performance  more  broadly,  intervening  in  detail 
only  in  particular  instances  where  malfunctioning  appears  to  be 
serious. 

2.  Congressional  review  would  tend  to  make  the  Foundation  less 
willing  to  fund  innovative  ideas. 

3.  Congressional  review  would  tend  to  politicize  the  award  decision- 
making process. 
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4.  Proper  Congressional  review  would  entail  a  great  deal  of  Con- 
gressional staff  effort  and  would  further  lengthen  the  time  it  takes 
applicants  to  obtain  funding  and  proceed  with  their  research. 

James  Symington: 

The  role  of  Congress  and  this  Committee  is  and  has  been  to  review  the  program 
and  budgets  of  the  executive  agencies.  Necessarily,  the  emphasis  in  this  process 
is  upon  the  major  objectives  of  managerial  performance  and  program  achieve- 
ment. In  my  view.  Congress  is  not  at  its  best  when  it  enters  into  detailed  review 
or  attempts  to  intervene  in  the  analysis  of  research  projects  where  scientific 
opinions  differ.  (878) 

William  Carey: 

I  would  regard  the  Bauman  amendment,  ...  as  a  very  serious  barrier  to  the 
type  of  evaluation  of  risk  and  value  in  risk-taking,  which  is  part  of  the  discovery 
process,  if  it  were  enacted  and  carried  out,  ...  I  think  the  whole  bureaucracy 
would  play  it  so  safe  and  so  placidly  and  cautiously  just  to  stay  out  of  the  rain  and 
the  thunder  and  the  liehtning  that  I  think  it  would  have  a  decidedly  braking  effect 
through  indirection.  (777) 

Charles  Mosher: 

Referring  to  Mr.  Bauman's  proposal,  I  would  merely  say  that  I  think  Congress- 
men would  be  far  more  timid  and  far  more  conscious  and  less  likely  to  support 
innovation,  particularly  radical  innovation,  than  the  scientific  community.  (28o) 

Thomas  Harkin : 

(P)hilosophers  have  dreamed  for  centuries  of  a  country,  of  a  nation,  where 
inquiry,  where  freedom  of  thought,  inquiry  of  any  nature  would  not  be  subject 
to  the  political  pressures  of  the  monarchy  or  dictator.  I  feel  whenever  inquiry 
and  freedom  of  thought  is  placed  beyond  the  realm  of  the  politician,  then  that 
country  acquires  a  spiritual  strength  that  makes  it,  in  fact,  a  beacon  of  liberty  in 
the  world.  That  is  what  I  want  to  protect.  I  don't  want  to  become  another  Soviet 
Union  or  China,  where  all  scientific  thought  is  dictated,  where  people  are  chan- 
nelized into  their  programs.  I  want  freedom  of — whatever  nature,  social  sciences, 
hard  sciences — freedom  to  inquire  and  to  challenge,  the  freedom  to  challenge 
the  most  deeply  rooted  belief  that  people  hold.  Only  by  doing  that  can  people 
really  change  and  grow. 

I  think  this  prior  approval  would  take  away  that  freedom  of  inquiry.  (240) 

James  Symington: 

I  think  whatever  kind  of  frying  pan  we  may  be  in  now  with  respect  to  oversight 
might  turn  into  a  fire  of  old  boyism  if  Congress  became  the  court  of  last  resort 
on  all  these  grants . . .  Little  by  little,  it  seems  to  me,  a  form  of  congressional  courtesy 
would  take  over  which  might  even  turn  into  an  active  club  on  the  NSF  to  go 
forward  with  grants  which  it  may  have  had  some  suspicion  about.  (241) 

As  it  pertains  to  the  Foundation's  decision-making  system  for 
individual  awards,  there  are  several  ways  in  which  Congressional 
oversight  should  be  carried  out.  Proper  execution  of  oversight  responsi- 
bilities requires  that  certain  data  be  available  to  various  arms  of 
Congress. 

There  was  strong  disagreement  between  Kep.  John  Conlan  and 
Foundation  witnesses  concerning  the  willingness  of  the  Foundation  to 
provide  information  to  Congress.  The  record  shows  the  facts  to  be  as 
follows : 

Xon-confidential  Information:  Unless  the  Foundation  has  im- 
plicitly or  explicitly  promised  a  source  that  it  will  keep  information 
confidential,  the  information  will  be  provided  promptly  upon  reason- 
able request  to  any  Member  of  Congress.  The  record  of  correspondence 
between  Rep.  Conlan  and  the  Foundation  contained  in  the  hearings 
record  is  ample  evidence. 
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Confidential  Information:  If  the  Foundation  has  promised  that 
information  will  be  kept  confidential,  it  will  be  made  available  on 
request  to  the  General  Accounting  Office,  the  House  Committee  on 
Science  and  Technology  or  its  Subcommittee  on  Science,  Research  and 
Technology,  and  to  the  Senate  Committee  on  Labor  and  Public 
Welfare  or  its  Subcommittee  on  the  National  Science  Foundation.  The 
Foundation  requests  that  any  arm  of  Congress  using  confidential 
information  provided  by  the  Foundation  take  steps  to  maintain  the 
highest  degree  of  confidentiality  of  the  information  compatible  with 
effective  oversight.  The  Foundation  has  refused,  however,  to  provide 
confidential  information  to  individual  Members  of  Congress.  The 
Subcommittee  believes  these  arrangements  for  the  treatment  of 
confidential  information  are  suitable  and  allow  for  effective  oversight 
of  the  Foundation  while  avoiding  disruption  of  the  Foundation's 
decision-making  processes. 

The  kind  of  confidential  information  that  has  heretofore  been  the 
principal  bone  of  contention  is  verbatim  peer  reviews  and  the  identifi- 
cation of  which  peers  reviewed  a  given  proposal.  Rep.  Conlan  has 
requested  and  been  denied  access  to  such  information.  The  GAO 
routinely  has  access  to  and  uses  it.  Neither  the  House  Committee  on 
Science  and  Technology  nor  its  Subcommittee  on  Science,  Research 
and  Technology  has  requested  access.1 

C.  INFORMATION  EXCHANGE  AND  APPEALS 

1.  Degree  of  Openness 

The  most  thoroughly  discussed  issue  of  the  hearings  was  this:  How 
far  can  openness  of  the  decision-making  process  coexist  with  the 
effective  evaluation  of  award  applications  by  the  National  Science 
Foundation? 

There  was  overwhelming  agreement  that  openness  per  se  is  desirable 
irrespective  of  its  effects  on  the  process  and  that  applications  ought  to 
be  evaluated  effectively.  The  extent  to  which  these  two  desiderata  are 
in  conflict,  and  which  should  be  favored  in  the  conflict  were  hotly 
debated. 

Charles  Mosher: 

I  am  certainly  one,  Mr.  Chairman,  who  has  always  believed  in  openness  as  the 
best  type  of  a  system  in  all  levels  of  government.  (30) 

James  Scheuer: 

Recently  we  have  discovered  what  a  cleansing  effect  openness  has.  It  seems  to 
cure  a  lot  of  problems.  (579) 

One  set  of  arguments  concerned  the  extent  to  which  the  comments 
and  identity  of  peer  reviewers  should  be  known.  The  alternatives  fall 
roughly  on  a  line  from  most  secret  to  most  open  as  follows :  commentary 
accompanies  the  viewpoints. 

1.  Applicants  should  know  nothing  about  who  reviewers  were  or 
what  they  said. 

No  one  ascribed  to  this  extreme  position. 

2.  Applicants  should  receive  paraphrased  reviewer  comments  on 
request  but  should  not  know  the  identity  of  reviewers. 

This  represented  Foundation  policy  for  many  years  until  it  was 
changed  in  June  1975  by  the  National  Science  Board.  The  new 
policy  is  given  by  item  (3).  It  took  effect  January  1,  1976. 

1  Bejrinninsr  in  1076,  the  subcommittee  has  examined  certain  confidential  documents  of  the 
National  Science  Foundation,  overtaking  publication  of  this  report. 
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3.  Applicants  should  receive  verbatim  reviewer  comments  on  request 
but  should  not  know  the  identity  of  reviewers. 

This  has  been  the  Foundation's  policy  since  January  1,  1976. 
Witnesses  who  agreed  that  this  option  or  the  preceding  one  is  best 
included:  Raymond  Bowers,  William  Carey,  Daniel  Drucker, 
Vincent  Haneman,  and  Donald  Marlowe.  Witnesses  whose 
opinion  lay  between  this  option  and  the  next  one  included: 
Philip  Morrison,  James  Powell,  and  Donald  Rice. 

4.  Applicants  should  receive  signed  verbatim  peer  reviews  on  re- 
quest. 

Rep.  John  Conlan  and  Heinz  Wilsdorf  believed  this  would  be 
best. 

5.  Applicants  and  the  public  at  large  should  have  access  to  signed 
verbatim  peer  reviews. 

No  one  claimed  this  would  be  a  good  policy. 

The  National  Science  Board  has  been  deliberating  the  confidentiality 
issue.  The  arguments  it  has  considered  encompass  all  the  major 
arguments  made  at  the  peer  review  hearings  and  were  elegantly  sum- 
marized by  Donald  Rice,  viz: 

Mr.  Chairman,  there  is,  as  you  are  well  aware,  probably  no  more  sensitive  issue 
in  the  award  decisionmaking  process  than  the  question  of  confidentiality.  I  can  say 
that  with  considerable  conviction  because  the  task  force  spent  nearly  all  of  its 
time  examining  the  pros  and  cons  of  that  one  subissue  alone  at  our  meeting  in 
June.  I  would  like  to  list  for  you  some  of  the  major  arguments  for  and  against 
confidentiality. 

Those  who  favor  confidentiality  in  some  degree  argue  one  or  more  of  the 
following: 

(1)  Reviewers  will  be  more  candid  on  all  aspects  of  the  proposal,  and  the  quali- 
fications of  the  proposer  to  accomplish  the  work,  if  their  identity  and/or  verbatim 
comments  are  kept  confidential  and  not  passed  on  to  the  proposer.  Thus,  con- 
fidentiality, in  this  view,  contributes  to  the  quality  of  reviews. 

(2)  Providing  names  of  peer  reviewers  to  principal  investigators  is  likely  to 
encourage  personality  clashes  between  the  proposer  and  the  reviewer,  leading  to 
unnecessary  ill  will  and  hard  feelings  among  all  parties  concerned. 

(3)  Program  managers  might  have  to  take  on  the  role  of  arbitrators  between 
proposers  and  reviewers,  and  would  also  find  themselves  spending  more  and  more 
of  their  time  defending  their  decisions  orally  and  in  writing.  This  would  add 
measurably  to  their  workload,  and  make  it  more  difficult  to  come  to  the  decisions 
they  have  to  reach. 

(4)  Many  potential  reviewers  would  refuse  to  participate  in  a  system  that 
required  them  to  be  identified  with  their  comments  to  the  authors  of  the  proposals 
they  have  reviewed. 

(5)  Unless  their  identity  was  protected,  reviewers  would  be  reluctant  to  com- 
ment critically  on  proposals  by  others  who  could  adversely  affect  the  career  path 
of  the  reviewers. 

(6)  The  identification  of  reviewers  would  increase  the  opportunity  for  political 
pressures  to  intrude  on  the  system.  I  understand,  Mr.  Chairman,  that  this  particu- 
lar problem  has  already  been  discussed,  as  have  others,  at  these  hearings. 

Those  favoring  an  open  system  argue  one  or  more  of  the  following: 

(1)  Qualified  reviewers  can  be  relied  upon  to  be  candid  and  straightforward  in 
their  evaluation  of  a  proposal,  regardless  of  whether  the  system  is  open  or  closed. 
Reviewers  who  are  not  willing  to  defend  their  positions  in  an  open  system  ought 
not  to  be  reviewing  Federal  grant  proposals  in  any  case. 

(2)  Openness  would  result  in  more  responsible  and  objective  reviews.  Superficial 
or  personality-based  comments  would  diminish,  and  the  result  would  be  a  more 
focused  and  effective  evaluation. 

(3)  Qualified  scientists  will  continue  to  participate  in  the  interest  of  furthering 
the  best  quality  science. 

(4)  Confidentiality  makes  the  system  unnecessarily  difficult  to  defend  from 
charges  of  internal  bias,  old-boy  network  practices,  favoritism,  or  other  criticisms. 
That  is,  an  open  system  can  better  demonstrate  its  impartiality  and  effectiveness.. 
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(5)  An  open  system  would  increase  the  workload  of  program  officers  in  some 
ways  and  reduce  it  in  others.  In  any  event,  staffing  levels  can  be  adjusted  to  the 
new  workload. 

(6)  Openness  means  change  and  change  may  cause  some  problems  initially; 
but,  in  the  long  run,  the  system  will  recalibrate  and  stronger  reviews  will  result. 
The  new  system  will  stand  public  scrutiny,  and  science  will  achieve  a  higher  level 
of  credibility.  (767,  768) 

These  arguments  are  directed  primarily  towards  making  a  decision 
between  position  3  and  position  4  from  the  list  of  five  viewpoints. 
Witnesses  generally  believed  that  the  Foundation's  new  policy, 
position  3,  will  be  an  improvement  over  the  old  one,  position  2,  but  the 
kind  of  information  presented  in  support  of  this  belief  and  in  support 
of  the  arguments  summarized  above  was  largely  anecdotal,  or  personal 
opinion,  or  a  small  opinion  survey. 

The  type  of  information  presented  concerning  two  critical  questions 
is  summarized  below. 

1.  What  will  happen  to  the  quality  of  reviews  if  signed  verbatim 
reviews  are  provided  to  applicants  (and,  possibly,  to  the  public)? 

Those  who  said  that  quality  would  either  stay  the  same  or  improve 
cited  only  their  own  personal  opinion. 

Those  who  said  that  quality  would  deteriorate  cited  their  own 
opinions  and: 

(a)  Vincent  Haneman  surveyed  172  engineers;  76  responded. 
Dr.  Haneman  reported  that  when  asked  "if  it  were  the  will  of 
Congress  that  public  identity  of  the  reviewer  be  published  along 
with  his  review,  .  .  .  (would)  more  candid,  honest  evalua- 
tions .  .  .  result  from  those  who  did  volunteer?"  Eight  said  yes 
and  64  indicated  less  candid,  less  honest  evaluations. 

(b)  The  analogy  was  drawn  between  peer  reviews  and  personal 
recommendations.  One  witness  pointed  out  that  on  his  campus 
the  contents  of  a  student's  personnel  file  were  available  for 
perusal  by  the  student.  This  had  resulted  in  student  evaluations 
by  professors  that  were  exclusively  positive. 

(2)  How  many  people  will  refuse  to  be  reviewers  if  signed  verbatim 
reviews  are  provided  to  applicants  (and,  possibly,  to  the  public)? 

Those  who  said  the  number  would  be  comparatively  small  cited 
only  their  own  opinions. 

Those  who  said  the  number  would  be  relatively  large  cited  their  own 
opinions  and: 

(a)  The  witness  who  pointed  out  the  analogy  to  personal 
recommendations  said  several  professors  had  refused  to  write 
evaluations  of  students  under  the  open  system. 

(b)  Of  Vincent  Haneman's  survey  respondents,  when  asked 
"would  the  faculty  still  be  available  to  provide  this  service?" 
Under  the  conditions  described  earlier,  27  said  yes  and  46  said  no. 

Additionally,  it  is  worth  noting  that  in  current  practice  the  NIH 
study  section  system  and  the  Foundation  ad  hoc  reviewing  system 
share  the  principle  that  reviewers'  names  should  be  separated  from 
their  opinions  in  material  provided  to  applicants,  in  that  the  Founda- 
tion will  be  providing  verbatim  comments  but  withholding  reviewers' 
names,  whereas  NIH  makes  names  available  but  maintains  separation 
by  paraphrasing  comments.  In  his  suggestions  for  review  at  the 
Foundation,  James  Powell  implied  that  if  individual  verbatim  reviews 
were  provided  to  applicants,  reviewer  names  should  be  withheld,  but 
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if  panels  are  employed,  names  should  be  provided.  There  is  a  decree 
of  realism  observable  in  current  practice  that  should  be  kept  in  mind, 
which  is  that  it  is  impossible  to  keep  secret  the  names  of  members  of 
a  standing  committee  who  travel  to  Washington  several  times  annu- 
ally—so to  be  equitable  they  should  be  published  so  that  the  entire 
community  will  be  informed,  not  just  a  favored  part  of  it.  On  the  other 
hand,  it  is  a  fairly  simple  matter  to  maintain  the  names  of  mail 
reviewers  in  confidentiality. 

As  the  following  resolution  shows,  the  National  Science  Board  has 
decided  to  make  available  the  names  of  peer  reviewers  in  an  aggregated 
form.  All  witnesses  who  mentioned  it  felt  this  was  a  step  in  the  right 
direction,  but  several  felt  that  the  reviewer  list  should  be  broken 
down  to  identify  the  reviewers  in  particular  Foundation  programs. 
The  list  of  reviewers  used  by  the  Foundation  in  1974  has  been  pre- 
pared by  the  Foundation  and  printed  by  the  Subcommittee.  The  level 
of  aggregation  of  future  lists  is  a  subject  of  continuing  debate.  One 
contribution  of  the  list  will  be  to  allow  for  the  recognition  of  those 
scientists  who  have  contributed  to  the  advance  of  science  through 
reviewing  applications  submitted  to  the  Foundation. 

RESOLUTION  ADOPTED   BY  THE   XATIOXAL   SCIENCE   BOARD  AT   ITS   174TH 
MEETING    ON    JUNE    20,    1975,    ON    PEER    REVIEW    INFORMATION 

The  National  Science  Board  has  examined  the  use  of  peer  review  in 
the  National  Science  Foundation  decision  process  on  grant  awards 
and  decimations.  The  Board  intends  the  peer  review  process  to  aid 
the  effective  evaluation  of  proposals  with  the  fairest  possible  treat- 
ment of  each  individual  proposal  and  the  broadest  possible  participa- 
tion of  qualified  scientists  and  other  appropriate  persons.  The  Board 
intends  that  the  review  process  be  conducted  with  as  much  openness 
and  information  to  proposers  as  possible  consistent  with  effective 
administration  of  the  decision  process.  To  these  ends  the  National 
Science  Board  Resolved  that: 

1.  The  Foundation  will  publish  annually  a  list  of  all  reviewers 
used  by  each  Division; 

2.  Program  officers  should  seek  broadly  representative  par- 
ticipation of  qualified  individuals  as  reviewers: 

3.  Verbatim  copies  of  reviews  requested  by  the  Foundation 
after  January  1,  1976.  not  including  the  identity  of  the  reviewer, 
will  be  made  available  to  the  principal  investigator  project  di- 
rector upon  request.  The  question  of  including  the  identity  of 
the  reviewer  will  be  considered  further  by  the  National  Science 
Board : 

4.  The  Foundation,  upon  request,  will  inform  the  principal 
investigator  project  director  of  the  reasons  for  its  decision  on  the 
proposal . 

All  reviews  requested  prior  to  January  1,  1976.  will  continue  to  be 
governed  by  earlier  policies  since  those  reviews  will  have  been  solicited 
with  a  commitment  on  the  part  of  the  National  Science  Foundation 
to  the  confidentiality  established  by  that  earlier  policy. 

The  Board  believes  this  new  policy  will  serve  to  improve  the  infor- 
mation exchange  with  the  scientific  community  and  allow  it  to  better 
understand  the  reasons  behind  National  Science  Foundation  decisions. 
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2.  Feedback  to  Applicants 

The  question  of  what  information  should  be  provided  to  applicants 
is  obviously  intertwined  with  the  confidentiality  issue  just  discussed. 
Excluding  the  matters  of  reviewers'  names  and  verbatim  comments, 
however,  there  was  general  agreement  among  witnesses  who  addressed 
feedback  that  proposers  should  have  the  right  to  know  the  rationale 
behind  the  decisions  taken  on  their  applications,  and  should  be  able 
to  get  it  in  writing.  Moreover,  proposers  should  be  informed  of  their 
right  to  feedback  as  a  matter  of  course;  it  was  not  clear  from  the 
testimony  whether  the  Foundation  has  adequately  notified  applicants 
in  the  past  concerning  what  information  they  may  have  on  request. 

3.  Appeals 

Neither  the  Foundation  nor  NIH  has  a  formal  procedure  whereby 
an  applicant  whose  proposal  has  been  rejected  can  have  the  same 
proposal  reconsidered.  A  rejected  applicant  is  expected  to  revise  and 
resubmit  the  proposal  if  a  new  review  is  desired.  Instances  of  revision 
and  resubmission  are  common  at  the  Foundation.  In  the  compara- 
tively rare  instances  of  allegations  from  proposers  that  the  Foundation 
should  reconsider  a  proposal,  the  Foundation  acts  in  an  ad  hoc  fashion, 
normally  at  least  checking  that  appropriate  procedures  were  used  in 
the  evaluation  of  the  proposal. 

The  consensus  of  witnesses  who  addressed  the  issue  of  appeals  was 
that  the  Foundation  should  have  an  established  policy  allowing  some 
form  of  appeal  by  a  rejected  applicant,  but  that  the  policy  should  not 
make  an  appeal  so  simple  that  applicants  would  overload  the  system 
with  appeals. 

A  variety  of  differing  appeal  procedures  were  suggested  by:  (1) 
Carlton  Hazlewood,  Gilbert  Ling,  and  Freeman  Cope,  (2)  Heinz 
Wilsdorf,  and  (3)  Rustum  Roy,  but  none  was  widely  endorsed.  None 
of  those  suggested  has  immediate  and  conspicuous  attractiveness  to 
the  Subcommittee.  Others  who  spoke  in  favor  of  appeals  (William 
Carey,  Donald  Marlowe,  James  Powell)  made  no  suggestions  for 
procedures. 

Arguments  for  appeals  are : 

1.  The  Foundation  is  not  perfect;  it  undoubtedly  makes  poor 
decisions  on  occasion.  To  be  fair  there  should  be  a  way  for  an 
aggrieved  applicant  who  has  been  rejected  in  a  poor  decision  to 
have  a  new  hearing. 

2.  If  peer  review  does  tend  to  reject  innovative  proposals,  since 
innovation  is  essential  to  the  progress  of  science,  a  procedure  is 
needed  to  check  peer  review  and  insure  that  important  innova- 
tions are  supported. 

Arguments  against  appeals  are: 

1.  It  is  recognized  that  many  proposals  of  sufficiently  high 
quality  to  have  been  funded  in  former  years  are  currently  being 
rejected  by  the  Foundation  because  of  funding  limitations.  If  an 
appeals  procedure  is  available  it  will  be  used  by  so  many  people 
that  an  inordinate  proportion  of  the  Foundation's  efforts  will  be 
devoted  to  handling  appeals. 

2.  Formal  appeal  procedures  will  introduce  adversary  relation- 
ships into  the  scientific  community  that  have  heretofore,  for- 
tunately been  missing.  If  there  are  frequent  or  conspicuous 
confrontations — between  reviewers  and  applicants,  in  particular — 
the  peer  review  system  will  be  jeopardized, 
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D.    DISTRIBUTION    OF    FUNDS 


1.  Geographical  Distribution 

Discussions  of  the  issue  of  distribution  of  funds  suffer  from  the 
fact  that  although  the  law  concerning  the  Foundation  enjoins  the 
Foundation  to  ''avoid  undue  concentration  of  such  research  and 
education,"  neither  the  Foundation  nor  Congress  has  attempted  to 
define  ''undue."  Judgements  of  the  Foundation's  performance  are 
consequently  very  subjective  and  open  to  disagreement. 

Of  the  various  distribution  issues,  geographical  distribution  was  the 
mo-t  thoroughly  discussed.  The  Foundation  produced  two  reports 
dealing  with  distribution  for  the  hearings,  "An  Analysis  of  the  Geo- 
graphical Distribution  of  XSF  Awards  as  Compared  with  Other 
Selected  Indicators"  and  "XSF  Management  Statistics." Both  reported 
data  for  Fiscal  Year  1974  and  are  included  in  the  record  with  the 
testimony  of  Richard  Atkinson.  A  copy  of  the  summary  data  sheet 
from  the  "Geographical  Distribution"  report  follows.  The  testimony 
of  Richard  Atkinson  dealt  explicitly  with  geographical  distribution, 
and  several  additional  graphs  were  presented  depicting  the  distribu- 
tion of  Foundation  funds.  Only  one  other  witness,  Doris  Wilsdorf, 
presented  geographical  distribution  data. 

Some  observations  concering  the  data  provided  by  the  Foundation 
follow.  The  Subcommittee  believes  these  observations  show  that  on  a 
Foundation-wide  basis  the  distribution  of  funds  is  at  least  not  grossly 
unfair.  The  other  information  available  to  the  Subcommittee  is  not  in 
dissonance  with  tins  conclusion.  The  aggregation  of  data  over  all 
Foundation  programs,  however,  ma-ks  the  distribution  of  funding  in 
any  given  program,  and  it  cannot  be  concluded  from  the  available 
information  that  the  distribution  in  any  program  is  fair  or  reasonable 
by  some  measure.  On  the  other  hand,  neither  was  any  data  presented 
to  the  subcommittee  which  demonstrated  unfair  distribution  of  fund- 
in  any  Foundation  program,  although  Prof.  Wilsdorf  purported  that 
her  data  indicated  "That  geographical  concentration  of  funding 
rather  than  dispersal  of  funding  has  taken  place." 

1.  The  success  ratios  of  the  51  states  (including  the  District  of 
Columbia),  vary  from  a  low  of  30  percent  (South  Carolina)  to  a  high  of 
60  percent  (District  of  Columbia).  The  success  ratio  is  the  proportion 
of  proposals  from  the  state  that  are  funded  in  some  amount  by  the 
Foundation.  The  best  success  ratio  is  thus  not  excessively  high  and  only 
twice  as  high  as  the  worst  success  ratio,  which  is  not  excessively  low. 

2.  Excluding  success  ratios,  the  rankings  on  the  summary  data  -heet 
of  states  by  ten  other  indicators  of  state  size  and  Foundation  awards 
show  that  most  states  are  ranked  similarly  by  all  indicators.  Forty 
states  have  at  least  nine  of  the  ten  indicators  lying  in  some  11-point 

nterval.  Connecticut,  for  instance,  has  rankings  of  17.  20.  19.  24.  18, 
22,  25,  22,  22,  and  16,  all  ten  of  which  fall  in  the  9-point  interval  from 
16  to  25. 

3.  The  dollar  amounts  of  Foundation  awards  per  science  or  engineer- 
ing doctorate  holder  on  the  other  hand  vary  widely.  Thirty-two 
states  received  between  $1,000  and  83,000  per  such  doctorate.  Nine 
received  less,  to  a  minimum  of  $500  (Idaho),  and  ten  received  more, 
to  a  maximum  of  $14,000  (West  Virginia).  Most  of  these  nine  are 
scientifically  unique  in  certain  respects  (Alaska.  Hawaii,  and  three 
states  of  modest  population  with  National  Centers  supported  by  the 
Foundation,  for  example). 


49 

The  variation  of  Foundation  funding  with  respect  to  population  is 
similarly  wide. 

The  Foundation's  policy  has  been  that  in  cases  where  applications 
are  of  substantially  equal  scientific  merit,  consideration  should  be 
given  to  geographical  distribution  in  the  decision  of  which  applications 
will  be  funded.  This  policy  has  had  the  advantage  of  avoiding  the 
"apples  and  oranges"  problem  of  weighing  geography  against  merit; 
merit  is  given  pr  macy,  and  geography  serves  to  help  make  fine 
distinctions. 

All  witnesses  who  addressed  the  question  of  how  geography  should 
be  factored  into  the  Foundation's  disbursement  of  funds  agreed  that 
peer  review  procedures  should  not  be  expected  to  encompass  geo- 
graphical considerations.  Several  of  those  witnesses  went  further  and 
said  that  if  Congress  finds  the  distribution  of  Foundation  funds  cur- 
rently lacking  in  some  respects,  Congress  should  establish  special 
programs  to  redress  the  imbalances  rather  than  attempting  to  adjust 
decision-making  procedures  for  present  programs  to  effect  a  redistri- 
bution. Charles  Kidd  explored  this  matter  in  some  detail. 

Raymond  Bowers: 

The  peer  review  process  is  largely  designed  to  maintain  quality.  But  quality 
of  scientific  work  is  not  the  only  factor  that  should  be  considered  in  the  allocation 
of  Federal  moneys.  Relevance  to  national  goals  and  considerations  of  geographical 
distribution  are  proper  concerns.  .  .  .  However,  I  would  not  look  to  the  peer 
review  system  of  itself  to  be  the  principal  administrative  device  for  achieving 
these  other  purposes.  They  require  the  design  of  special  programs  and  special 
procedures.  (466) 

Charles  Kidd: 

The  peer  review  system  carries  out  basic  goals  set  by  Congress  and  in  greater 
detail  by  the  Federal  agencies.  Those  who  deplore  concentration  of  funds  and 
concentration  on  support  of  science  should  attack  the  basic  policies  and  programs 
themselves  and  not  the  peer  review  system.  (475)    ■ 

In  the  past  Congressional  concern  for  the  nationwide  strengthening 
of  science  was  such  that  there  were  special  programs  devoted  to  that 
purpose.  These  have  now  been  largely  shut  down.  Within  the  Sub- 
committee there  was  substantial  disagreement  concerning  the  em- 
phasis that  should  be  placed  on  geographical  distribution. 

Rep.  James  Scheuer  said  the  Foundation  should  use  Federal  funds 
to  support  excellence,  wherever  and  in  whatever  concentration  it 
may  be  found.  Rep.  Walter  Flowers,  contrariwise,  said  he  thought  the 
Foundation  was  not  heeding  the  provision  to  "avoid  undue  concen- 
tration" and  said  there  was  a  great  deal  of  room  for  improvement. 

It  must  be  observed,  moreover,  that  the  Foundation,  while  the  main 
supporter  of  many  fields  of  basic  science,  provides  only  a  small  fraction 
of  the  total  support  for  scientific  research  and  education  in  the  United 
States.  If  there  is  or  is  to  be  a  national  policy  concerning  the  geo- 
graphical distribution  of  funds  for  science,  it  will  have  to  encompass  at 
least  those  agencies  which  provide  the  bulk  of  the  funds  in  order  to  be 
effective.  Requiring  even  geographical  distribution  of  the  Foundation 
funds  only  will  not  achieve  the  result  which  might  be  desired  of 
assuring  geographically  broad  performance  of  research. 

The  effect  of  multiple  funding  sources  on  the  distribution  of  funds 
is  intensified  when  a  single  specialty  is  considered.  To  examine  Founda- 
tion support  of  atomic  physics  without  concurrently  investigating 
ERDA  support,  for  example,  would  result  in  a  thoroughly  distorted 
evaluation  of  whether  the  levels  of  support  for  the  subdisciplines  of 
atomic  physics  by  the  Foundation  are  reasonable. 
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2.  Institutional  Distribution 

The  distribution  of  Foundation  funds  among  academic,  profit,  non- 
profit, government,  and  other  institutions  was  not  addressed  to  any 
significant  degree  in  the  peer  review  hearings  (one  letter  in  the  record 
broaches  the  subject  of  the  funding  of  for-profit  institutions).  This  is 
a  subject  that  may  deserve  further  attention  at  another  time. 

The  preponderance  of  Foundation  funds  (about  77%  in  Fiscal  Year 
1974)  is  awarded  to  academic  institutions,  however,  and  there  was 
some  discussion  of  whether  the  division  of  Foundation  funds  among 
various  types  of  academic  institutions  is  equitable.  James  Powell 
argued  strongly  that  basic  research  awards  are  too  often  denied  to 
proposers  from  colleges  (without  graduate  departments).  The  statistics 
show  that  only  about  one  percent  of  the  total  Foundation  funds 
directed  to  academic  institutions  go  to  private,  bachelors  degree  only 
institutions. 

It  has  been  frequently  alleged  in  the  past  that  peer  review  systems 
fail  to  support  good  scientists  at  lesser  institutions.  Two  arguments 
why  this  may  be  true  are:  (a)  peer  reviewers  are  less  likely  to  personally 
know  scientists  at  lesser  institutions,  and  (b)  the  peer  reviewers' 
perceptions  of  individual  quality  may  be  colored  by  perceptions  of 
institutional  quality — in  other  words  the  institution  rubs  off  on  the 
individual.  If  these  two  effects  are  indeed  strong,  then  it  may  be  that 
the  Foundation  is  getting  less  for  its  money  than  it  would  if  colleges 
and  less  prestigious  universities  were  more  heavily  supported.  Founda- 
tion witnesses  claimed,  to  the  contrary,  that  scientists  in  minor 
institutions  would  receive  at  least  an  even  break. 

But,  there  are  good  reasons  why  the  Foundation  should  fund  less 
research  at  colleges  and  less  prestigious  universities,  and  it  is  extremely 
difficult  not  to  confound  the  effects  of  the  good  reasons  below  with 
those  of  the  bad  ones  above. 

Good  reasons:  (a)  the  better  research  scientists  tend  to  congregate 
in  the  better  universities,  as  mentioned  earlier;  (b)  well-to-do  uni- 
versities frequently  have  experimental  facilities  and  apparatus  not  found 
at  other  institutions;  the  proposer  from  the  better  equipped  institution 
is  more  likely  to  succeed  in  the  experiment  and,  therefore,  has  a 
justified  edge  on  the  competition;  (c)  teaching  loads  at  colleges 
(commonly  12  class  hours  per  week)  allow  the  college  professor  much 
less  time  for  research  than  the  university  professor  with  a  much 
lighter  teaching  load  (commonly  6  or  9  class  hours  per  week) ;  (d)  at 
universities,  graduate  students  are  available  to  do  much  of  the  routine 
research  work;  this  provides  training  for  future  practicing  scientists; 
if  the  same  research  were  performed  at  colleges  it  may  be  difficult  to 
find  assistance  of  the  same  caliber,  and  the  training  effects  might  be 
dissipated  on  persons  not  planning  scientific  careers. 

The  last  argument  may  be  turned  around  with  some  cogency: 
undergraduates  are  more  enthusiastic,  and,  if  they  are  the  future 
graduate  students,  will  not  dissipate  the  training.  Moreover,  the  type 
of  baccalaureate-granting  institution  whose  undergraduate  students 
are  most  likely  to  go  on  to  earn  a  doctorate  is  known  to  be  the  good 
private  college. 
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3.  Young  Scientists 

Those  witnesses  who  mentioned  age  agreed  that  it  is  vital  to  support 
the  best  members  of  each  new  generation  of  scientists.  Younger 
scientists  are  potentially  liable  to  be  at  a  disadvantage  because  they 
have  little  or  no  past  record  by  which  the  quality  of  their  future 
performance  may  be  estimated.  Foundation  witnesses  said  that  this 
would  be  taken  into  account  by  the  system;  younger  investigators 
would  be  given  a  chance  to  prove  themselves.  No  witness  claimed 
that  younger  investigators  have  been  underfunded,  although  Carlton 
Hazlewood  said  he  had  heard  complaints  about  peer  review  from 
younger  scientists.  However,  no  data  whatsoever  were  presented  to 
the  Subcommittee  showing  how  younger  investigators  have  actually 
fared  in  recent  years. 

Guy  ford  Stever: 

XSF  is  increasingly  concerned  with  the  entry  of  younger  scientists  into  Founda- 
tion support  competition.  (141) 

Richard  Atkinson: 

The  process  is  anything  but  biased  in  favor  of  better-known  investigators,  in 
fact  it  may  be  biased  in  favor  of  persons  who  first  appear  before  the  review 
process.  (281) 

William  Carey: 

(We)  have  very  little  information  that  will  tell  us  whether  peer  review  has  had 
good  or  adverse  effects  upon  the  chances  of  younger  scientists  to  get  on  with  their 
pioneering  ...  I  am  suggesting  that  it  would  be  a  good  thing  if  peer  review 
could  be  made  into  a  lively  process  for  discovering  and  encouraging  young  talent 
in  science  as  an  explicit  function.  The  NSF  seems  confident  that  this  is  going  on.  I 
will  only  say  that  it  needs  to  be  demonstrated,  not  merely  asserted.  (779) 

The  current  employment  crisis  in  science  introduces  a  new  difficulty 
into  the  support  of  younger  scientists.  In  many  fields  of  science  the 
size  of  the  national  faculty  grew  rapidly  during  the  1950's  and  1960's 
but  has  ceased  growing  in  the  seventies.  This  means  that  each  year 
there  are  few  positions  to  fill  with  new  PhD's — especially  at  the  best 
institutions  where,  it  has  been  observed,  a  higher  proportion  of  the 
faculty  is  tenured  than  in  other  institutions.  At  the  same  time,  large 
numbers  of  new  doctorates  are  being  awarded  by  the  nation's  uni- 
versities. The  result  is  that  many  very  competent  new  PhD's  will 
not  be  employed  by  the  best  institutions.  Whereas  in  the  past  a  very 
significant  proportion  of  the  best  young  scientists  could  be  found  at 
the  most  prestigious  institutions,  the  proportion  will  likely  be  rela- 
tively small  in  the  future.  The  Foundation  may  find  it  necessary  to 
take  special  steps  to  insure  the  rest  of  the  good  younger  scientists  have 
the  opportunity  to  do  research. 

E.    OTHER 

1.  National  Science  Foundation  Programs 

Three  specific  Foundation  programs  were  addressed  by  different 
sets  of  witnesses:  materials  research  (Doris  Wilsdorf),  engineering 
(Donald  Marlowe  and  Daniel  Dracker),  and  biochemistry  (Carlton 
Hazlewood.  Freeman  Cope,  and  Gilbert  Ling). 

Donald  Marlowe  and  Daniel  Drucker  proposed  that  funding  of 
engineering  research  by  the  Foundation  is  inadequate.  The  Subcom- 
mittee has  not  attempted  to  analyze  this  contention  in  the  present 
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report.  As  mentioned  at  the  outset,  the  hearings  did  not  bear  on  the 
issue  of  priority  setting  among  scientific  fields  and  purposes.  This  is 
an  area  of  such  magnitude  and  importance  it  needs  and  deserves 
separate,  thorough  investigation.  If  the  Subcommittee,  or  another 
body,  does  investigate  Foundation  priority  setting  at  some  later  date, 
it  would  be  appropriate  to  include  the  issue  of  support  of  engineering. 

Doctors  Hazlewood,  Cope,  and  Ling  used  a  biochemical  theory 
issue  to  stress  the  need  for  a  better  method  of  insuring  that  innovative 
research  is  funded.  This  need  has  been  dealt  with  in  earlier  sections  of 
this  report  concerning  innovative  research  and  appeals.  The  Subcom- 
mittee does  not  choose,  however,  to  attempt  to  judge  the  scientific 
issue  involved  because  the  theory  in  question  has  been  funded  by 
another  agency  (NIH)  and  because  Subcommittee  members  believe 
such  judgement  lies  outside  their  competence.  For  the  latter  reason  the 
Subcommittee  has  also  not  attempted  to  judge  the  scientific  perform- 
ance of  those  Foundation  staff  members  concerned  with  this  matter. 

The  materials  research  testimony  of  Doris  Wilsdorf  has  received 
wide  attention  since  the  peer  review  hearings.  Professor  Wilsdorf 
maintained  that  her  survey  and  analysis  of  funding  by  the  Founda- 
tion's Metallurgy  and  Materials  Section  showed  that  certain  uni- 
versity departments  were  being  discriminated  against  by  the  Section 
in  such  a  way  that  some  of  the  best  researchers  in  the  field  were  being 
underfunded  by  the  Section.  Her  methodology  relied  heavily  on  cita- 
tion analysis.  Because  Doris  Wilsdorfs  testimony  was  quite  technical, 
the  Subcommittee  had  it  analyzed  by  the  Science  Policy  Division  of 
the  Congressional  Research  Service.  About  twenty  members  of  the 
metallurgy  and  materials  scientific  community  have  written  to  the 
Subcommittee  concerning  the  testimony.  The  Research  Service 
analysis  and  eight  of  the  letters  are  included  in  Appendix  II  of  the 
hearings  record.  The  analysis  and  all  correspondents  are  unanimous  in 
their  judgement  that  the  methodology  of  the  testimony  is  faulty. 
The  primary  objection  is  that  citation  analysis  standing  alone  is 
inadequate  to  produce  a  true  measure  of  the  ability  of  a  scientist. 
A  second  objection  is  that,  to  be  meaningful,  an  analysis  of  metallurgy 
and  materials  funding  patterns  must  encompass  all  agencies  providing 
support,  not  only  the  Foundation.  Additionally,  many  issues  con- 
cerning technical  aspects  of  the  evidence  presented  by  Professor 
Wilsdorf  have  been  raised.  All  correspondents  who  mentioned  the 
Foundation  staff  said  they  believe  the  staff  is  competent  and  unbiased. 
It  was  not  clear,  however,  whether  the  Foundation  had  adequately 
filled  requests  by  the  principal  witness  for  information. 

2.  Overlap  oj  Research 

The  National  Science  Foundation  is  not,  of  course,  the  only  Federal 
agency  which  supports  basic  research.  There  are  scientific  fields  sup- 
ported by  the  Foundation  and  by  one  or  more  other  agencies.  It  is 
inevitable  that  there  will  be  some  projects  funded  by  the  Foundation 
which  might  equally  well  be  supported  by  other  agencies.  Indeed, 
the  Foundation  jointly  funds  quite  a  number  of  projects  with  other 
agencies.  The  question  to  be  addressed  is,  to  what  extent  is  overlap  of 
Foundation  program  areas  with  other  agency  program  areas  desirable. 

One  witness,  Rep.  Robert  Bauman,  felt  that  such  overlaps  lead  to 
the  Foundation  "duplicating  the  work  of  other  agencies"  and  should  bft 
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eliminated  insofar  as  possible.  Other  witnesses  disagreed,  saying  that 
the  fact  there  have  been  several  prospective  sources  of  funding  for 
most  scientific  proposals  has  been  one  of  the  great  strengths  of  Amer- 
ican science.  If,  as  Guyford  Stever  said,  the  support  of  basic  scientific 
research  is  nearly  a  Federal  monopoly,  it  may  be  essential  not  to  leave 
the  nurture  of  any  scientific  field  entirely  in  the  hands  of  a  single 
agency.  Eliminating  overlaps  may  be  a  threat  to  innovation. 

In  any  event,  the  issue  is  one  involving  all  Federal  agencies,  not  the 
Foundation  alone,  and  if  and  when  it  is  addressed  should  probably  be 
addressed  on  a  government-wide  basis. 

Rustum  Roy: 

Meanwhile,  the  great  American  invention,  of  the  multiple  sources  of  funding, 
which  was  the  truly  American  innovation,  that,  I  think,  is  endangered.  (754,  755) 

Richard  Atkinson: 

The  National  Science  Board  has  stated  the  policy  and  I  am  in  agreement  with 
it;  namely,  that  one  agency  should  not  be  responsible  for  all  basic  research,  that 
researchers  should  be  able  to  seek  support  from  several  agencies.  This  insures  a 
certain  variation  of  opinion  about  what  is  worth  funding,  and  what  isn't,  and 
over  time  if  one  agency  has  missed  the  boat  and  funded  the  wrong  line  of  research, 
another  agency  will  hopefully  have  been  correct  in  identifying  the  right  one.  (284) 


VI.  Findings  and  Recommendations 

A.  SUPPORT  OF  HIGH-QUALITY  RESEARCH 

1.  Finding:  Procedures  relying  on  peer  review  for  evaluating  applications  for 
scientific  research  awards  can  lead  to  the  effective  advance  of  scientific  knowledge. 
The  National  Science  Foundation's  peer  review  evaluation  systems  appear  basi- 
cally sound. 

No  method  superior  to  peer  review  has  been  found  for  judging  the 
scientific  merit  of  basic  research  proposals  and  the  scientific  com- 
petence of  proposers.  Appropriate  peer  review  procedures  generally 
lead  to  the  support  of  proposals  in  a  high  quality  range. 

Recommendation:  The  National  Science  Foundation  should  continue  to  use 
some  form  of  peer  review.  Congress  should  not  require  major  changes  at  this  time 
in  the  methods  used  by  the  Foundation  for  proposal  evaluation. 

2.  Finding:  Certain  problems  exist  with  the  Foundation's  peer  review  evalua- 
tion processes  and  related  operations.  Substantial  changes  in  the  operation  of 
the  Foundation's  peer  review  systems  have  been  proposed  which  possibly  could 
lead  to  improvements  in  the  systems'  effectiveness  for  consistently  selecting  the 
best  proposals,  and  in  the  acceptability  of  the  systems  to  potential  applicants, 
the  scientific  community,  and  Congress. 

Recommendation:  Some  corrective  actions  should  be  taken  immediately  and 
certain  proposals  for  substantial  changes  should  be  thoroughly  investigated  and 
instituted  if  judged  worthy. 

Details  of  the  corrective  actions  and  studies  of  proposed  substantial 
changes  are  discussed  in  succeeding  findings  and  recommendations. 

3.  Finding:  Proposals  for  truly  innovative  research,  which  is  essential  to  the 
progress  of  science,  may  possibly  not  be  adequately  funded  through  peer  review 
selection  procedures. 

Recommendation:  The  National  Science  Board  should  study  the  support  of 
innovative  research  and  report  to  Congress. 

The  study  should  include  at  least:  (1)  a  detailed  analysis  of  the 
issues,  starting  with  those  covered  in  this  report,  (2)  a  retrospective 
look  at  some  major  scientific  innovations — with  emphasis  on  this 
century — how  they  were  funded  and  how  they  came  to  be  accepted, 
(3)  an  assessment  of  the  adequacy  of  decision-making  procedures 
used  by  National  Science  Foundation  programs  for  finding  and  fund- 
ing innovative  research,  and  (4)  recommendations,  if  necessary,  for 
modified  procedures  or  new  programs  designed  to  ensure  that  innova- 
tive research  is  funded. 

B.  MANAGING  PEER  REVIEW  AT  THE  FOUNDATION 

1.  Finding:  There  is  a  clear  need  for  firm  policy  guidance  in  the  management 
of  peer  review  at  the  National  Science  Foundation.  Experience  in  science  and  with 
the  scientific  community  as  well  as  carefully  assembled  objective  information  are 
essential  to  the  formation  of  sound  policies  governing  peer  review. 

Recommendation:  The  National  Science  Board  should  have  primary  respon- 
sibility for  the  establishment  of  policies  governing  peer  review  at  the  National 
Science  Foundation. 

(55) 
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In  this  recommendation  the  Subcommittee  reaffirms  the  Board's 
legislated  charge  from  Congress  to  set  Foundation  policies.  The 
Subcommittee  is  aware  that  the  Board  has  been  examining  Founda- 
tion policies  vigorously  in  recent  months  and  applauds  this  activity. 
Because  of  the  Subcommittee's  responsibilities  for  oversight  and  legis- 
lation concerning  the  Foundation,  it  is  desirable  that  the  Board 
turn  its  attention  promptly  to  the  Subcommittee's  recommendations 
for  Board  actions. 

2.  Finding:  There  may  be  serious  future  problems  with  the  Foundation's 
operation  caused  by  the  large  number  of  competent  scientists  competing  for 
limited  funds. 

These  potential  problems  include  general  dissatisfaction  with  the 
Foundation  on  the  part  of  the  scientific  community,  overloads  on  the 
Foundation  staff  causing  poor  proposal  evaluation,  and  missed 
opportunities  caused  by  good  scientists  dropping  out  of  the  race  for 
funds.  The  overload  problem  may  already  be  significant. 

Recommendation:  The  National  Science  Board  should  establish  an  internal 
Foundation  program  to  monitor  problems  arising  from  the  mismatch  between  the 
size  of  the  scientific  community  and  the  amount  of  Foundation  funds  available 
for  support  of  that  community,  and  should  report  periodically  to  Congress. 

Initial  reports  should  deal  with  what  is  being  monitored  and  why, 
and  with  the  overload  pioblem. 

3.  Finding:  The  communication  of  Foundation  policies  within  the  Foundation 
and  with  the  scientific  communitj^  at  large  should  be  improved. 

Examples  of  staff  actions  contrary  to  Foundation  policy  were  given 
at  the  healings.  Although  the  Subcommittee  does  not  believe  such 
actions  are  widespread,  the  compendium  of  policy  and  procedure 
memoranda  submitted  by  the  Foundation,  because  of  its  size  and  lack 
of  organization,  gives  the  Subcommittee  little  reassurance  that  the 
staff  can  stay  abreast  of  current  policies  even  with  the  best  intentions. 
The  Subcommittee  believes  that  the  scientific  community  should 
be  able  to  understand  and  stay  current  with  Foundation  policies.  At 
present  there  is  no  way  such  understanding  can  be  easily  acquired. 

Recommendation:  The  Foundation  should  prepare  an  organized  compendium 
of  its  policies  in  writing  and  assure  that  it  be  widely  available. 

The  form  and  content  of  the  compendium  should  have  the  approval 
of  the  National  Science  Board. 

The  Subcommittee  suggests,  without  being  prescriptive,  that:  (1) 
information  about  policies  be  separated  from  information  about  pro- 
cedures; procedural  information  is  less  important  and  apt  to  change 
more  quickly,  it  need  not  be  in  the  compendium;  (2)  the  compendium 
should  be  in  a  form  that  may  be  easily  updated;  and  (3)  copies  of  the 
compendium  should  be  available  to  all  Foundation  staff  members  and 
to  the  public. 

4.  Finding:  Performance  of  the  staff  of  the  National  Science  Foundation  has 
been  a  key  determinant  of  the  Foundation's  performance  in  the  evaluation  of 
proposals.  The  Subcommittee  believes  the  staff  is  generally  dedicated,  competent, 
and  honest. 

There  have  been  instances  of  staff  mistakes  and  failures  of  judgment, 
but  the  Subcommittee  is  largely  of  the  opinion  that  these  are  as 
infrequent  and  isolated  as  any  human  endeavor  in  a  large  organization 
will  allow. 
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Recommendation:  Because  of  the  central  importance  of  the  staff,  the  potential 
effects  on  staff  of  major  changes  in  the  Foundation's  decision-making  processes  for 

individual  awards  should  be  explicitly  considered  before  such  changes  are  made. 

5.  Finding:  The  reviewing  of  proposals  by  persons  outside  the  Foundation 
should  be  distributed  widely  among  competent  potential  reviewers. 

This  is  already  Foundation  policy,  and,  on  a  Foundation-wide 
geographic  basis,  rather  broad  distribution  appears  to  have  oc- 
curred. There  are,  however,  no  systematic  methods  employed  b}^  the 
Foundation  for  assuring  wide  distribution  or  determining  whether 
any  given  reviewer  has  been  too  frequently  used. 

Recommendation:  The  Foundation  should  devise  systematic  methods  of  ensur- 
ing that  competent  reviewers  are  identified  and  used  regardless  of  institutional 
affiliation  or  geographical  location.  No  reviewer  should  be  overused  in  relation  to 
his  or  her  abilities. 

The  Foundation's  management  information  system,  the  Subcom- 
mittee understands,  has  (or  can  readily  be  modified  to  have)  informa- 
tion in  it  concerning  reviewers  which  should  make  the  execution  of 
this  recommendation  a  fairly  straight  fo  ward  matter  not  adding 
appreciably  to  the  staff  workload. 

6.  Finding:  Strong  arguments  have  been  put  forward  for  more  extensive 
Foundation  use  of  peer  panel  review. 

Increased  reliance  on  peer  panel  review  was  the  most  convincingly 
supported  major  change  proposed  for  the  Foundation's  peer  review 
systems. 

Recommendation:  The  National  Science  Board  should  study  the  extent  to 
which  the  Foundation  should  rely  on  peer  panel  review  and  report  to  Congress. 

The  Subcommittee  itself  does  not  feel  it  has  sufficient  information  or 
experience  with  the  operation  of  peer  review  systems  to  come  now  to 
a  definitive  conclusion  on  this  matter.  The  Subcommittee  does  believe, 
however,  that  the  National  Science  Board  should  give  the  increased 
use  of  peer  panels  very  serious  consideration. 

As  part  of  its  study  the  Board  should  determine  the  feelings  of  the 
scientific  community. 

7.  Finding:  The  philosophy  of  policing  the  National  Science  Foundation's  peer 
review  system  should  be  based  on  trust  of  the  Foundation's  staff  and  reviewers, 
combined  with  vigilance  over  the  system  to  assure  that  trust  is  warranted. 

The  Subcommittee  observes  that: 

(a)  Peer  review  systems  based  on  trust  allow  more  latitude  for 
staff  discretion  and,  hence,  allow  better  performance  than  those 
based  on  rigid,  highly  structured  procedures. 

(b)  A  staff  which  is  not  trusted  is  unlikely  to  take  pride  in  its 
work  and  will  therefore  perform  less  well  than  a  trusted  staff. 

(c)  Reviewers  may  drop  out  of  the  peer  review  system  if  they 
feel  they  are  not  trusted. 

(d)  A  peer  review  system  replete  with  a  variety  of  checking 
mechanisms  is  apt  to  be  inefficient,  although  no  system  for  the 
allocation  of  funds  is  likely  to  operate  properly  in  the  absence  of 
checks. 

(e)  The  public  is  not  well  served  if  the  oversight  of  a  Federal 
granting  system  relies  on  the  absence  of  indications  that  things 
are  going  wrong  rather  than  the  presence  of  indications  that  the 
system  is  working  properly. 

Recommendation:  The  National  Science  Foundation  should  perform  ran- 
dom audits  of  the  decision  processes  used  for  individual  grants.  The  Foundation 
should  report  to  Congress  describing  the  system  it  uses  for  this  type  of  audit. 
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These  audits  should  be  performed  in  sufficient  detail  to  address 
questions  of  relations  among  Foundation  staff  members,  applicants, 
and  reviewers.  Members  of  the  auditing  staff  should  not  normally  have 
responsibilities  in  the  process  of  decision-making  for  individual  awards. 
The  audits  will  not  replace  any  activities  of  the  General  Accounting 
Office  or  other  Congressional  arms.  Further  aspects  of  vigilance  over 
operation  of  the  Foundation's  peer  review  system  are  described  in 
other  findings  and  recommendations. 

8.  Finding:  Congressional  review  of  individual  National  Science  Foundation 
awards  would  introduce  serious  problems  into  the  government's  support  of  science. 

The  problems  include : 

(a)  Congressional  review  would  tend  to  politicize  the  award 
decision-making  process  and  turn  it  awa}'  from  the  support  of 
proposals  on  their  merit. 

(b)  Congressional  review  would  tend  to  make  the  Foundation 
less  willing  to  fund  innovative  ideas. 

(c)  Congressional  review  would  further  lengthen  the  time  it- 
takes  applicants  to  obtain  funding  and  proceed  with  their  research. 

Recommendation:  Congress  should  not  require  Congressional  review  of  individ- 
ual research  awards. 

9.  Finding:  It  may  be  necessary  to  proper  Congressional  oversight  for  the 
Subcommittee  on  Science,  Research  and  Technology  of  the  House  Committee 
on  Science  and  Technology  and  the  Subcommittee  on  the  National  Science 
Foundation  of  the  Senate  Committee  on  Labor  and  Public  Welfare  to  examine 
certain  confidential  information  of  the  Foundation. 

Such  confidential  information  would  include  peer  reviews  and  the 
identification  of  which  peers  reviewed  a  given  proposal.  Although  the 
General  Accounting  Office  has  had  access  to  and  used  such  informa- 
tion, to  settle  future  controversies  of  the  type  that  were  raised  at  the 
hearings,  these  two  Subcommittees  may  need  to  examine  such  infor- 
mation firsthand. 

Recommendation:  These  House  and  Senate  Subcommittees  should  work  with 
the  Foundation  to  establish  procedures  whereby  the  Subcommittees  can  have 
effective  access  to  certain  confidential  information  of  the  Foundation.  The  pro- 
cedures should  be  designed  to  maintain  the  highest  degree  of  confidentiality  of 
the  information  that  will  allow  effective  Congressional  oversight. 

The  Director  of  the  Foundation  has  already  indicated  his  willingness 
that  this  recommendation  be  implemented. 

C.    INFORMATION    EXCHANGE    AND    APPEALS 

1.  Finding:  Openness  of  Foundation  operations  and  the  effective  evaluation 
of  proposals  by  the  Foundation  are  both  desirable. 

Recommendation:  The  greatest  degree  of  openness  of  the  Foundation's  award 
decision-making  process  consonant  with  effective  proposal  evaluation  and  reason- 
able efficiency  should  be  achieved. 

2.  Finding:  In  keeping  with  the  policy  of  maximum  feasible  openness,  the 
recent  decision  of  the  National  Science  Board  to  publish  a  list  of  reviewers  used 
by  the  Foundation  is  laudable. 

Recommendation:  The  National  Science  Board  should  study  the  effects  of 
publication  of  the  list  and  consider  whether  publication  of  the  list  in  a  less  aggre- 
gated form  might  be  desirable. 

3.  Finding:  Information  concerning  the  critical  problem  of  effects  on  the 
Foundation's  peer  review  system  of  the  level  of  confidentiality  in  which  peer 
reviewer's  names  and  verbatim  comments  are  held  is  inadequate  for  making  sound 
judgements  on  what  that  level  of  confidentiality  should  be. 
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Recommendation:  The  National  Science  Board  should  collect  further  informa- 
tion concerning  effects  on  the  peer  review  system  of  the  level  of  confidentiality 
in  which  peer  reviewers  names  and  verbatim  comments  are  held.  The  Hoard 
Should  report  the  information  and  any  conclusions  that  may  he  drawn  from  it  to 
Congress.  Further  changes  in  the  level  of  confidentiality  of  the  Foundation's 
peer  review  system  should  he  made  slowly  if  at  all. 

The  Subcommittee  observes  that  there  appears  to  be  no  immediate 
need  for  changes  in  the  level  of  confidentiality.  The  Board's  resolution 
of  June  1975,  changing  this  level  took  effect  January  1,  1976.  The 
Board's  information  collection  efforts  should  include  attempts  to 
obtain  quantitative  data  (the  fi action  of  requested  reviews  that  get 
done,  average  time  between  request  and  receipts  of  reviews,  etc.) 
which  will  illuminate  effects  of  the  change.  The  Board  should  also 
collect  opinions  from  the  scientific  community  about  the  confidentiality 
question.  The  Subcommittee  understands  the  Board  has  already  begun 
collecting  the  latter  type  of  data. 

4.  Finding:  Each  applicant  for  a  National  Science  Foundation  award  should 
have  easy  access  to  a  description  of  the  rationale  for  the  decision  taken  on  the 
application  and  a  description  of  the  decision-making  process  itself. 

It  is  not  clear  that  applicants  have  had  adequate  access  to  such 
information  in  the  past. 

Recommendation:  The  Foundation  should  include  in  each  letter  announcing  a 
funding  decision  to  an  applicant  either  a  statement  of  the  methods  and  rationale  of 
the  decision  or  a  statement  that  such  information  will  be  provided  on  request. 

5.  Finding:  Strong  arguments  have  been  made  both  for  and  against  have  in  a 
formal  method  for  making  appeals  to  decisions  on  award  applications  at  the 
National  Science  Foundation. 

Recommendation:  The  National  Science  Board  should  study  the  question  of 
whether  the  National  Science  Foundation  should  have  formal  procedures  for 
considering  appeals  of  decisions  made  on  award  applications  and  should  report  to 
Congress. 

This  study  should  be  coordinated  with  the  study  of  the  support  of 
innovative  research  recommended  earlier.  None  of  the  specific  appeal 
procedures  proposed  by  witnesses  displayed  compelling  characteristics 
to  the  Subcommittee  on  a  first  examination. 

D.    DISTRIBUTION    OF    FUNDS 

1.  Finding:  There  is  division  of  opinion  among  members  of  the  Subcommittee 
concerning  the  desirability  of  requiring  that  the  geographical  distribution  of 
National  Science  Foundation  funds  meet  some  standard  of  evenness.  Moreover, 
questions  of  the  geographical  distribution  of  Federal  funds  for  scientific  research 
should  be  considered  on  a  government-wide  basis,  including  all  agencies  providing 
significant  amounts  of  money  for  scientific  research,  not  only  the  Foundation. 

Recommendation:  If  Congress  undertakes  studies  of  the  geographical  distri- 
bution of  funds  for  scientific  research  by  any  Federal  agency  or  agencies,  such 
studies  should  explicitly  put  support  by  one  agency  in  the  context  of  total  Federal 
support. 

The  Subcommittee  is,  nonetheless,  mindful  of  the  requirements  of 
Public  Law  81-507,  Section  3(e),  the  organic  Act  of  the  Foundation, 
and  cautions  the  Foundation  that  it  has  special  statutory  duty  to 
avoid  undue  concentration  in  its  grant  decisions. 

2.  Finding:  There  are  indications,  not  contradicted  by  available  information, 
that  research  at  undergraduate-teaching  institutions  without  graduate  depart- 
ments (colleges)  may  not  be  adequately  funded  by  the  National  Science  Foundation. 
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This  finding  applies  to  reseaich  awards  of  the  Foundation  only. 
There  do  not  appear  to  be  problems  with  the  overall  distribution  of 
education  awards  by  the  Foundation  to  colleges. 

Recommendation:  The  National  Science  Board  should  study  the  funding  of 
research  at  colleges  by  the  Foundation  and  report  to  Congress. 

The  study  should  provide  data  on  the  distribution  of  Foundation 
research  funds  to  colleges,  analyze  the  data  to  determine  the  suitability 
of  the  distribution,  and  give  conclusions  and  recommendations  for 
Foundation  procedures,  as  indicated.  This  study  should  be  coordinated 
with  the  recommended  monitoring  by  the  Foundation  of  problems 
arising  from  the  mismatch  between  the  size  of  the  scientific  community 
and  the  amount  of  Foundation  funds  available  for  support  of  that 
community. 

3.  Finding:  Although  it  is  widely  acknowledged  that  the  best  scientists  in  each 
generation  should  be  identified  and  funded  whenever  possible  and  that  emplojr- 
ment  problems  for  young  scientists  are  currently  severe  (and  likely  to  remain  so 
in  the  foreseeable  future),  there  is  little  information  available  about  the  distribu- 
tion to  young  scientists  of  funds  for  scientific  research  or  about  the  effects  on 
science  of  the  difficulties  of  young  scientists. 

Recommendation:  The  National  Science  Board  should  study  the  support  of 
young  scientists  and  report  to  Congress. 

The  study  should  be  coordinated  with  the  recommended  program  to 
monitor  difficulties  caused  by  the  mismatch  between  scientists  and 
funds.  The  study  should  include  quantitative  measures  of  the  support 
of  young  scientists  by  the  Foundation  over  a  period  of  time. 

E.    OTHER 

1.  Finding:  Evidence  presented  at  the  hearings  which  purported  to  show  that 
the  Foundation's  Metallurgy  and  Materials  Section  has  distributed  funds  poorly 
is  highly  questionable. 

The  primal  reasons  that  the  evidence  is  questionable  are:  (1)  cita- 
tion analysis  alone  was  used  to  produce  a  measure  of  scientific  ability, 
but  citation  analysis  standing  alone  appears  to  be  inadequate  for  that 
purpose,  and  (2)  to  be  meaningful,  an  analysis  of  funding  patterns 
must  encompass  all  agencies  providing  support.  Additionally,  many 
issues  concerning  technical  aspects  of  the  evidence  have  been  raised. 
It  was  not  clear,  however,  whether  the  Foundation  had  adequately 
filled  requests  by  the  principal  witness  for  information.  The  following 
recommendation  addresses  that  point. 

Recommendation:  The  Foundation  should  fill  reasonable  requests  for  data 
concerning  the  distribution  of  funds  by  the  Metallurgy  and  Materials  Section 
within  the  constraints  of  the  Foundation's  policies  concerning  the  confidentiality 
of  such  information. 

2.  Finding:  The  extent  to  which  the  fields  of  scientific  research  supported  by 
various  Federal  agencies  should  be  allowed  to  overlap  is  open  to  question. 

Arguments  are  strong  that  the  overlap  should  be  rather  extensive. 

Recommendation:  Congress  should  not  propose  or  acquiesce  to  reductions 
in  the  overlap  of  fields  of  basic  scientific  research  supported  by  various  Federal 
agencies  without  further  investigating  the  advantages  and  disadvantages  of  such 
overlaps. 
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